Jul 14 18:08:07.315374 ip-10-0-136-25 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 14 18:08:07.315386 ip-10-0-136-25 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 14 18:08:07.315396 ip-10-0-136-25 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 14 18:08:07.315732 ip-10-0-136-25 systemd[1]: Failed to start Kubernetes Kubelet. Jul 14 18:08:18.646288 ip-10-0-136-25 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 14 18:08:18.646305 ip-10-0-136-25 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot b05067e0551145bc907b9c2287b69fca -- Jul 14 18:09:55.041378 ip-10-0-136-25 systemd[1]: Starting Kubernetes Kubelet... Jul 14 18:09:55.480949 ip-10-0-136-25 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 18:09:55.480949 ip-10-0-136-25 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 14 18:09:55.480949 ip-10-0-136-25 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 18:09:55.480949 ip-10-0-136-25 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 14 18:09:55.480949 ip-10-0-136-25 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 18:09:55.483258 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.483121 2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 14 18:09:55.490762 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490738 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:09:55.490762 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490759 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:09:55.490762 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490764 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:09:55.490762 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490768 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:09:55.490996 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490773 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:09:55.490996 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490779 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:09:55.490996 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490783 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:09:55.490996 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490787 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:09:55.490996 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490791 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:09:55.490996 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490794 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:09:55.490996 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490797 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:09:55.490996 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490804 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:09:55.490996 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490811 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:09:55.490996 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490815 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:09:55.490996 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490819 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:09:55.490996 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490823 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:09:55.490996 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490827 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:09:55.490996 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490830 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:09:55.490996 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490834 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:09:55.490996 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490838 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:09:55.490996 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490842 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:09:55.490996 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490845 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:09:55.490996 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490849 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:09:55.490996 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490853 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:09:55.491870 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490857 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:09:55.491870 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490860 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:09:55.491870 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490864 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:09:55.491870 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490868 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:09:55.491870 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490872 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:09:55.491870 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490878 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:09:55.491870 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490883 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:09:55.491870 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490889 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:09:55.491870 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490893 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:09:55.491870 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490898 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:09:55.491870 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490902 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:09:55.491870 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490906 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:09:55.491870 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490910 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:09:55.491870 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490915 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:09:55.491870 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490920 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:09:55.491870 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490924 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:09:55.491870 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490928 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:09:55.491870 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490932 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:09:55.491870 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490936 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:09:55.491870 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490940 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:09:55.492821 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490945 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:09:55.492821 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490949 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:09:55.492821 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490953 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:09:55.492821 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490957 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:09:55.492821 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490961 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:09:55.492821 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490966 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:09:55.492821 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490970 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:09:55.492821 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490975 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:09:55.492821 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490979 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:09:55.492821 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490983 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:09:55.492821 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490987 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:09:55.492821 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490992 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:09:55.492821 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490996 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:09:55.492821 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.490999 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:09:55.492821 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491003 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:09:55.492821 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491008 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:09:55.492821 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491012 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:09:55.492821 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491016 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:09:55.492821 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491020 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:09:55.492821 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491025 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:09:55.493662 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491032 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:09:55.493662 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491037 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:09:55.493662 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491041 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:09:55.493662 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491046 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:09:55.493662 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491050 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:09:55.493662 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491055 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:09:55.493662 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491059 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:09:55.493662 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491063 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:09:55.493662 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491068 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:09:55.493662 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491072 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:09:55.493662 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491077 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:09:55.493662 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491081 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:09:55.493662 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491085 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:09:55.493662 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491090 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:09:55.493662 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491113 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:09:55.493662 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491119 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:09:55.493662 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491123 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:09:55.493662 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491130 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:09:55.493662 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491134 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:09:55.494467 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491139 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:09:55.494467 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491144 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:09:55.494467 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491150 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:09:55.494467 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491156 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:09:55.494467 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491160 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:09:55.494467 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491165 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:09:55.494467 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491169 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:09:55.494467 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491174 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:09:55.494467 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491178 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:09:55.494467 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491182 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:09:55.494467 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491186 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:09:55.494467 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491190 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:09:55.494467 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491194 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:09:55.494467 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491198 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:09:55.494467 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491202 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:09:55.494467 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.491206 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:09:55.494467 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492445 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:09:55.494467 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492457 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:09:55.494467 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492463 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:09:55.495307 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492468 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:09:55.495307 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492473 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:09:55.495307 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492477 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:09:55.495307 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492482 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:09:55.495307 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492486 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:09:55.495307 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492490 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:09:55.495307 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492495 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:09:55.495307 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492499 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:09:55.495307 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492503 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:09:55.495307 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492507 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:09:55.495307 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492511 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:09:55.495307 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492514 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:09:55.495307 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492519 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:09:55.495307 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492523 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:09:55.495307 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492532 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:09:55.495307 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492537 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:09:55.495307 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492541 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:09:55.495307 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492545 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:09:55.495307 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492550 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:09:55.495307 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492554 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:09:55.495914 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492560 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:09:55.495914 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492566 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:09:55.495914 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492571 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:09:55.495914 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492576 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:09:55.495914 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492580 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:09:55.495914 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492585 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:09:55.495914 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492589 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:09:55.495914 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492594 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:09:55.495914 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492598 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:09:55.495914 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492603 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:09:55.495914 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492607 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:09:55.495914 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492612 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:09:55.495914 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492617 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:09:55.495914 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492621 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:09:55.495914 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492625 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:09:55.495914 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492629 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:09:55.495914 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492636 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:09:55.495914 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492642 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:09:55.495914 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492646 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:09:55.496528 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492651 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:09:55.496528 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492656 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:09:55.496528 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492661 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:09:55.496528 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492665 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:09:55.496528 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492669 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:09:55.496528 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492673 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:09:55.496528 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492677 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:09:55.496528 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492681 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:09:55.496528 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492686 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:09:55.496528 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492691 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:09:55.496528 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492696 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:09:55.496528 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492700 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:09:55.496528 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492703 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:09:55.496528 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492707 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:09:55.496528 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492712 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:09:55.496528 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492718 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:09:55.496528 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492724 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:09:55.496528 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492728 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:09:55.496528 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492732 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:09:55.497027 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492736 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:09:55.497027 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492740 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:09:55.497027 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492743 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:09:55.497027 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492748 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:09:55.497027 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492752 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:09:55.497027 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492756 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:09:55.497027 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492760 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:09:55.497027 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492764 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:09:55.497027 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492770 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:09:55.497027 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492774 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:09:55.497027 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492778 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:09:55.497027 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492783 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:09:55.497027 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492787 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:09:55.497027 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492791 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:09:55.497027 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492795 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:09:55.497027 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492799 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:09:55.497027 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492804 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:09:55.497027 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492808 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:09:55.497027 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492812 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:09:55.497027 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492816 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:09:55.497703 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492820 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:09:55.497703 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492824 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:09:55.497703 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492828 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:09:55.497703 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492832 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:09:55.497703 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492836 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:09:55.497703 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492840 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:09:55.497703 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492844 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:09:55.497703 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492848 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:09:55.497703 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492852 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:09:55.497703 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492856 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:09:55.497703 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492860 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:09:55.497703 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492864 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:09:55.497703 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492868 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:09:55.497703 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492873 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:09:55.497703 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492876 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:09:55.497703 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492880 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:09:55.497703 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492884 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:09:55.497703 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.492888 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:09:55.497703 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493509 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:09:55.498178 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493519 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:09:55.498178 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493523 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:09:55.498178 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493527 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:09:55.498178 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493532 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:09:55.498178 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493536 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:09:55.498178 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493540 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:09:55.498178 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493544 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:09:55.498178 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493547 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:09:55.498178 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493552 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:09:55.498178 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493556 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:09:55.498178 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493560 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:09:55.498178 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493564 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:09:55.498178 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493568 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:09:55.498178 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493575 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:09:55.498178 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493581 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:09:55.498178 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493586 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:09:55.498178 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493592 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:09:55.498178 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493597 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:09:55.498178 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493602 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:09:55.498178 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493607 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:09:55.498717 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493611 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:09:55.498717 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493616 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:09:55.498717 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493621 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:09:55.498717 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493626 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:09:55.498717 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493630 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:09:55.498717 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493634 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:09:55.498717 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493638 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:09:55.498717 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493642 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:09:55.498717 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493646 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:09:55.498717 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493650 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:09:55.498717 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493655 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:09:55.498717 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493658 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:09:55.498717 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493663 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:09:55.498717 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493667 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:09:55.498717 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493671 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:09:55.498717 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493674 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:09:55.498717 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493678 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:09:55.498717 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493683 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:09:55.498717 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493687 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:09:55.499203 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493691 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:09:55.499203 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493696 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:09:55.499203 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493700 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:09:55.499203 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493705 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:09:55.499203 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493709 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:09:55.499203 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493713 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:09:55.499203 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493718 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:09:55.499203 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493722 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:09:55.499203 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493726 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:09:55.499203 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493732 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:09:55.499203 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493736 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:09:55.499203 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493740 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:09:55.499203 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493744 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:09:55.499203 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493748 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:09:55.499203 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493752 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:09:55.499203 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493756 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:09:55.499203 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493760 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:09:55.499203 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493767 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:09:55.499203 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493772 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:09:55.499203 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493777 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:09:55.499737 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493781 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:09:55.499737 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493786 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:09:55.499737 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493790 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:09:55.499737 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493795 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:09:55.499737 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493801 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:09:55.499737 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493808 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:09:55.499737 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493812 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:09:55.499737 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493816 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:09:55.499737 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493820 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:09:55.499737 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493825 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:09:55.499737 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493829 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:09:55.499737 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493833 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:09:55.499737 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493838 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:09:55.499737 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493842 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:09:55.499737 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493846 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:09:55.499737 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493850 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:09:55.499737 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493854 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:09:55.499737 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493858 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:09:55.499737 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493863 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:09:55.499737 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493867 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:09:55.500232 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493871 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:09:55.500232 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493875 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:09:55.500232 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493880 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:09:55.500232 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493884 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:09:55.500232 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493887 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:09:55.500232 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493891 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:09:55.500232 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493895 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:09:55.500232 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493899 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:09:55.500232 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493903 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:09:55.500232 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493907 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:09:55.500232 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493911 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:09:55.500232 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493914 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:09:55.500232 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493918 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:09:55.500232 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493922 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:09:55.500232 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493926 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:09:55.500232 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493930 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:09:55.500232 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493934 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:09:55.500232 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493941 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:09:55.500232 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.493945 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:09:55.500232 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494043 2565 flags.go:64] FLAG: --address="0.0.0.0" Jul 14 18:09:55.500720 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494068 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 14 18:09:55.500720 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494115 2565 flags.go:64] FLAG: --anonymous-auth="true" Jul 14 18:09:55.500720 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494124 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 14 18:09:55.500720 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494131 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 14 18:09:55.500720 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494136 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 14 18:09:55.500720 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494143 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 14 18:09:55.500720 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494149 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 14 18:09:55.500720 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494154 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 14 18:09:55.500720 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494158 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 14 18:09:55.500720 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494164 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 14 18:09:55.500720 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494170 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 14 18:09:55.500720 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494175 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 14 18:09:55.500720 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494179 2565 flags.go:64] FLAG: --cgroup-root="" Jul 14 18:09:55.500720 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494183 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 14 18:09:55.500720 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494188 2565 flags.go:64] FLAG: --client-ca-file="" Jul 14 18:09:55.500720 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494193 2565 flags.go:64] FLAG: --cloud-provider="external" Jul 14 18:09:55.500720 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494198 2565 flags.go:64] FLAG: --cluster-dns="[]" Jul 14 18:09:55.500720 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494204 2565 flags.go:64] FLAG: --cluster-domain="" Jul 14 18:09:55.500720 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494208 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 14 18:09:55.500720 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494213 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 14 18:09:55.500720 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494218 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 14 18:09:55.500720 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494223 2565 flags.go:64] FLAG: --container-log-max-files="5" Jul 14 18:09:55.500720 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494228 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 14 18:09:55.500720 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494233 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 14 18:09:55.501286 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494238 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 14 18:09:55.501286 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494243 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 14 18:09:55.501286 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494248 2565 flags.go:64] FLAG: --contention-profiling="false" Jul 14 18:09:55.501286 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494253 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 14 18:09:55.501286 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494257 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 14 18:09:55.501286 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494264 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 14 18:09:55.501286 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494269 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 14 18:09:55.501286 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494275 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 14 18:09:55.501286 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494280 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 14 18:09:55.501286 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494284 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 14 18:09:55.501286 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494289 2565 flags.go:64] FLAG: --enable-load-reader="false" Jul 14 18:09:55.501286 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494293 2565 flags.go:64] FLAG: --enable-server="true" Jul 14 18:09:55.501286 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494298 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 14 18:09:55.501286 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494305 2565 flags.go:64] FLAG: --event-burst="100" Jul 14 18:09:55.501286 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494310 2565 flags.go:64] FLAG: --event-qps="50" Jul 14 18:09:55.501286 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494314 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 14 18:09:55.501286 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494319 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 14 18:09:55.501286 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494324 2565 flags.go:64] FLAG: --eviction-hard="" Jul 14 18:09:55.501286 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494330 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 14 18:09:55.501286 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494334 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 14 18:09:55.501286 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494339 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 14 18:09:55.501286 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494344 2565 flags.go:64] FLAG: --eviction-soft="" Jul 14 18:09:55.501286 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494348 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 14 18:09:55.501286 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494352 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 14 18:09:55.501286 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494358 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 14 18:09:55.501966 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494363 2565 flags.go:64] FLAG: --experimental-mounter-path="" Jul 14 18:09:55.501966 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494367 2565 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 14 18:09:55.501966 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494372 2565 flags.go:64] FLAG: --fail-swap-on="true" Jul 14 18:09:55.501966 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494377 2565 flags.go:64] FLAG: --feature-gates="" Jul 14 18:09:55.501966 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494383 2565 flags.go:64] FLAG: --file-check-frequency="20s" Jul 14 18:09:55.501966 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494387 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 14 18:09:55.501966 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494392 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 14 18:09:55.501966 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494396 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 14 18:09:55.501966 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494401 2565 flags.go:64] FLAG: --healthz-port="10248" Jul 14 18:09:55.501966 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494406 2565 flags.go:64] FLAG: --help="false" Jul 14 18:09:55.501966 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494410 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-136-25.ec2.internal" Jul 14 18:09:55.501966 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494415 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 14 18:09:55.501966 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494421 2565 flags.go:64] FLAG: --http-check-frequency="20s" Jul 14 18:09:55.501966 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494426 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 14 18:09:55.501966 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494431 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 14 18:09:55.501966 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494437 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 14 18:09:55.501966 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494441 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 14 18:09:55.501966 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494446 2565 flags.go:64] FLAG: --image-service-endpoint="" Jul 14 18:09:55.501966 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494450 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 14 18:09:55.501966 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494455 2565 flags.go:64] FLAG: --kube-api-burst="100" Jul 14 18:09:55.501966 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494459 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 14 18:09:55.501966 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494464 2565 flags.go:64] FLAG: --kube-api-qps="50" Jul 14 18:09:55.501966 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494468 2565 flags.go:64] FLAG: --kube-reserved="" Jul 14 18:09:55.501966 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494472 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 14 18:09:55.502598 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494476 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 14 18:09:55.502598 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494481 2565 flags.go:64] FLAG: --kubelet-cgroups="" Jul 14 18:09:55.502598 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494485 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 14 18:09:55.502598 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494490 2565 flags.go:64] FLAG: --lock-file="" Jul 14 18:09:55.502598 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494494 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 14 18:09:55.502598 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494498 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 14 18:09:55.502598 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494503 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 14 18:09:55.502598 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494512 2565 flags.go:64] FLAG: --log-json-split-stream="false" Jul 14 18:09:55.502598 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494517 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 14 18:09:55.502598 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494523 2565 flags.go:64] FLAG: --log-text-split-stream="false" Jul 14 18:09:55.502598 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494527 2565 flags.go:64] FLAG: --logging-format="text" Jul 14 18:09:55.502598 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494532 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 14 18:09:55.502598 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494537 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 14 18:09:55.502598 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494541 2565 flags.go:64] FLAG: --manifest-url="" Jul 14 18:09:55.502598 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494546 2565 flags.go:64] FLAG: --manifest-url-header="" Jul 14 18:09:55.502598 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494552 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 14 18:09:55.502598 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494557 2565 flags.go:64] FLAG: --max-open-files="1000000" Jul 14 18:09:55.502598 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494563 2565 flags.go:64] FLAG: --max-pods="110" Jul 14 18:09:55.502598 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494568 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 14 18:09:55.502598 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494572 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 14 18:09:55.502598 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494577 2565 flags.go:64] FLAG: --memory-manager-policy="None" Jul 14 18:09:55.502598 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494581 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 14 18:09:55.502598 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494586 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 14 18:09:55.502598 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494590 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 14 18:09:55.502598 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494595 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 14 18:09:55.503199 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494606 2565 flags.go:64] FLAG: --node-status-max-images="50" Jul 14 18:09:55.503199 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494610 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 14 18:09:55.503199 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494615 2565 flags.go:64] FLAG: --oom-score-adj="-999" Jul 14 18:09:55.503199 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494619 2565 flags.go:64] FLAG: --pod-cidr="" Jul 14 18:09:55.503199 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494624 2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 14 18:09:55.503199 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494632 2565 flags.go:64] FLAG: --pod-manifest-path="" Jul 14 18:09:55.503199 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494636 2565 flags.go:64] FLAG: --pod-max-pids="-1" Jul 14 18:09:55.503199 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494640 2565 flags.go:64] FLAG: --pods-per-core="0" Jul 14 18:09:55.503199 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494647 2565 flags.go:64] FLAG: --port="10250" Jul 14 18:09:55.503199 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494651 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 14 18:09:55.503199 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494656 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06c579d791e11c5ed" Jul 14 18:09:55.503199 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494661 2565 flags.go:64] FLAG: --qos-reserved="" Jul 14 18:09:55.503199 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494665 2565 flags.go:64] FLAG: --read-only-port="10255" Jul 14 18:09:55.503199 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494670 2565 flags.go:64] FLAG: --register-node="true" Jul 14 18:09:55.503199 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494675 2565 flags.go:64] FLAG: --register-with-taints="" Jul 14 18:09:55.503199 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494680 2565 flags.go:64] FLAG: --registry-burst="10" Jul 14 18:09:55.503199 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494685 2565 flags.go:64] FLAG: --registry-qps="5" Jul 14 18:09:55.503199 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494690 2565 flags.go:64] FLAG: --reserved-cpus="" Jul 14 18:09:55.503199 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494694 2565 flags.go:64] FLAG: --reserved-memory="" Jul 14 18:09:55.503199 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494704 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 14 18:09:55.503199 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494709 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 14 18:09:55.503199 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494714 2565 flags.go:64] FLAG: --rotate-certificates="false" Jul 14 18:09:55.503199 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494719 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 14 18:09:55.503199 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494723 2565 flags.go:64] FLAG: --runonce="false" Jul 14 18:09:55.503199 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494728 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 14 18:09:55.503826 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494732 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 14 18:09:55.503826 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494737 2565 flags.go:64] FLAG: --seccomp-default="false" Jul 14 18:09:55.503826 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494742 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 14 18:09:55.503826 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494747 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 14 18:09:55.503826 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494752 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 14 18:09:55.503826 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494756 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 14 18:09:55.503826 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494761 2565 flags.go:64] FLAG: --storage-driver-password="root" Jul 14 18:09:55.503826 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494766 2565 flags.go:64] FLAG: --storage-driver-secure="false" Jul 14 18:09:55.503826 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494770 2565 flags.go:64] FLAG: --storage-driver-table="stats" Jul 14 18:09:55.503826 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494775 2565 flags.go:64] FLAG: --storage-driver-user="root" Jul 14 18:09:55.503826 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494779 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 14 18:09:55.503826 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494784 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 14 18:09:55.503826 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494788 2565 flags.go:64] FLAG: --system-cgroups="" Jul 14 18:09:55.503826 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494792 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 14 18:09:55.503826 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494800 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 14 18:09:55.503826 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494806 2565 flags.go:64] FLAG: --tls-cert-file="" Jul 14 18:09:55.503826 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494810 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 14 18:09:55.503826 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494816 2565 flags.go:64] FLAG: --tls-min-version="" Jul 14 18:09:55.503826 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494821 2565 flags.go:64] FLAG: --tls-private-key-file="" Jul 14 18:09:55.503826 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494825 2565 flags.go:64] FLAG: --topology-manager-policy="none" Jul 14 18:09:55.503826 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494829 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 14 18:09:55.503826 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494834 2565 flags.go:64] FLAG: --topology-manager-scope="container" Jul 14 18:09:55.503826 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494838 2565 flags.go:64] FLAG: --v="2" Jul 14 18:09:55.503826 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494844 2565 flags.go:64] FLAG: --version="false" Jul 14 18:09:55.503826 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494851 2565 flags.go:64] FLAG: --vmodule="" Jul 14 18:09:55.504461 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494857 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 14 18:09:55.504461 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.494863 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 14 18:09:55.504461 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495047 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:09:55.504461 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495054 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:09:55.504461 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495059 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:09:55.504461 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495063 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:09:55.504461 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495068 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:09:55.504461 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495072 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:09:55.504461 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495077 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:09:55.504461 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495081 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:09:55.504461 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495085 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:09:55.504461 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495091 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:09:55.504461 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495115 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:09:55.504461 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495120 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:09:55.504461 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495124 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:09:55.504461 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495127 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:09:55.504461 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495132 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:09:55.504461 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495136 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:09:55.504461 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495140 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:09:55.504461 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495143 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:09:55.504950 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495147 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:09:55.504950 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495151 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:09:55.504950 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495157 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:09:55.504950 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495161 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:09:55.504950 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495165 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:09:55.504950 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495169 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:09:55.504950 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495173 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:09:55.504950 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495176 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:09:55.504950 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495181 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:09:55.504950 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495185 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:09:55.504950 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495188 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:09:55.504950 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495193 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:09:55.504950 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495198 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:09:55.504950 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495202 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:09:55.504950 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495206 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:09:55.504950 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495210 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:09:55.504950 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495214 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:09:55.504950 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495219 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:09:55.504950 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495224 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:09:55.504950 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495228 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:09:55.505441 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495233 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:09:55.505441 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495238 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:09:55.505441 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495242 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:09:55.505441 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495246 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:09:55.505441 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495250 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:09:55.505441 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495254 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:09:55.505441 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495258 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:09:55.505441 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495262 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:09:55.505441 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495266 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:09:55.505441 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495270 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:09:55.505441 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495274 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:09:55.505441 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495278 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:09:55.505441 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495282 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:09:55.505441 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495286 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:09:55.505441 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495300 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:09:55.505441 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495304 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:09:55.505441 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495310 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:09:55.505441 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495315 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:09:55.505441 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495320 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:09:55.505911 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495342 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:09:55.505911 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495346 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:09:55.505911 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495350 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:09:55.505911 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495355 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:09:55.505911 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495359 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:09:55.505911 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495363 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:09:55.505911 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495369 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:09:55.505911 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495373 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:09:55.505911 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495377 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:09:55.505911 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495381 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:09:55.505911 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495386 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:09:55.505911 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495390 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:09:55.505911 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495394 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:09:55.505911 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495398 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:09:55.505911 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495406 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:09:55.505911 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495410 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:09:55.505911 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495416 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:09:55.505911 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495422 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:09:55.505911 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495427 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:09:55.506387 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495431 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:09:55.506387 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495435 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:09:55.506387 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495440 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:09:55.506387 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495445 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:09:55.506387 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495449 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:09:55.506387 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495453 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:09:55.506387 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495457 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:09:55.506387 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495462 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:09:55.506387 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495468 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:09:55.506387 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495472 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:09:55.506387 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495477 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:09:55.506387 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495481 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:09:55.506387 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495485 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:09:55.506387 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495489 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:09:55.506387 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495493 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:09:55.506387 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495497 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:09:55.506387 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495502 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:09:55.506387 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495506 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:09:55.506387 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495510 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:09:55.506827 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495514 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:09:55.506827 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495519 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:09:55.506827 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495523 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:09:55.506827 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.495527 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:09:55.506827 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.496239 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 18:09:55.506827 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.503641 2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 14 18:09:55.506827 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.503658 2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 14 18:09:55.506827 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.504661 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 14 18:09:55.506827 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.504669 2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 14 18:09:55.506827 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504724 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:09:55.506827 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504729 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:09:55.506827 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504733 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:09:55.506827 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504736 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:09:55.506827 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504739 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:09:55.506827 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504742 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:09:55.507206 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504745 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:09:55.507206 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504748 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:09:55.507206 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504751 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:09:55.507206 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504754 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:09:55.507206 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504757 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:09:55.507206 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504760 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:09:55.507206 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504763 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:09:55.507206 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504766 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:09:55.507206 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504769 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:09:55.507206 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504772 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:09:55.507206 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504775 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:09:55.507206 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504777 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:09:55.507206 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504780 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:09:55.507206 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504783 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:09:55.507206 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504785 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:09:55.507206 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504788 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:09:55.507206 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504790 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:09:55.507206 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504793 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:09:55.507206 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504795 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:09:55.507206 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504798 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:09:55.507690 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504800 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:09:55.507690 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504803 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:09:55.507690 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504807 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:09:55.507690 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504811 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:09:55.507690 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504815 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:09:55.507690 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504818 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:09:55.507690 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504820 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:09:55.507690 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504823 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:09:55.507690 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504825 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:09:55.507690 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504828 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:09:55.507690 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504830 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:09:55.507690 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504833 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:09:55.507690 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504835 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:09:55.507690 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504838 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:09:55.507690 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504840 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:09:55.507690 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504842 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:09:55.507690 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504845 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:09:55.507690 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504847 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:09:55.507690 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504849 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:09:55.508191 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504852 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:09:55.508191 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504854 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:09:55.508191 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504857 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:09:55.508191 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504859 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:09:55.508191 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504861 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:09:55.508191 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504864 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:09:55.508191 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504866 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:09:55.508191 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504868 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:09:55.508191 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504871 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:09:55.508191 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504873 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:09:55.508191 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504876 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:09:55.508191 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504878 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:09:55.508191 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504880 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:09:55.508191 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504883 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:09:55.508191 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504885 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:09:55.508191 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504888 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:09:55.508191 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504891 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:09:55.508191 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504894 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:09:55.508191 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504896 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:09:55.508191 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504899 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:09:55.508684 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504902 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:09:55.508684 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504904 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:09:55.508684 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504907 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:09:55.508684 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504909 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:09:55.508684 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504911 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:09:55.508684 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504914 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:09:55.508684 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504916 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:09:55.508684 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504918 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:09:55.508684 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504921 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:09:55.508684 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504923 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:09:55.508684 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504927 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:09:55.508684 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504930 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:09:55.508684 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504933 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:09:55.508684 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504935 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:09:55.508684 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504938 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:09:55.508684 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504941 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:09:55.508684 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504943 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:09:55.508684 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504945 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:09:55.508684 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504948 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:09:55.509180 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504950 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:09:55.509180 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504953 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:09:55.509180 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504955 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:09:55.509180 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504958 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:09:55.509180 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504960 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:09:55.509180 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504963 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:09:55.509180 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504965 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:09:55.509180 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504968 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:09:55.509180 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504971 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:09:55.509180 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504974 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:09:55.509180 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504977 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:09:55.509180 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504981 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:09:55.509180 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504984 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:09:55.509180 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504987 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:09:55.509180 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.504990 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:09:55.509539 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.504995 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 18:09:55.509539 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505119 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:09:55.509539 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505124 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:09:55.509539 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505127 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:09:55.509539 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505130 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:09:55.509539 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505132 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:09:55.509539 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505135 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:09:55.509539 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505138 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:09:55.509539 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505141 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:09:55.509539 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505144 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:09:55.509539 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505146 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:09:55.509539 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505148 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:09:55.509539 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505152 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:09:55.509539 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505155 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:09:55.509539 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505158 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:09:55.509892 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505160 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:09:55.509892 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505162 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:09:55.509892 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505165 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:09:55.509892 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505168 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:09:55.509892 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505170 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:09:55.509892 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505173 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:09:55.509892 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505175 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:09:55.509892 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505178 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:09:55.509892 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505180 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:09:55.509892 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505183 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:09:55.509892 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505187 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:09:55.509892 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505191 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:09:55.509892 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505194 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:09:55.509892 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505197 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:09:55.509892 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505201 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:09:55.509892 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505203 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:09:55.509892 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505206 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:09:55.509892 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505208 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:09:55.509892 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505211 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:09:55.510423 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505213 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:09:55.510423 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505216 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:09:55.510423 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505218 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:09:55.510423 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505220 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:09:55.510423 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505223 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:09:55.510423 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505225 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:09:55.510423 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505227 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:09:55.510423 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505230 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:09:55.510423 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505232 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:09:55.510423 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505234 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:09:55.510423 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505237 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:09:55.510423 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505239 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:09:55.510423 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505241 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:09:55.510423 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505244 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:09:55.510423 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505246 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:09:55.510423 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505250 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:09:55.510423 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505253 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:09:55.510423 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505255 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:09:55.510423 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505258 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:09:55.510423 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505261 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:09:55.510878 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505263 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:09:55.510878 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505265 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:09:55.510878 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505268 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:09:55.510878 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505270 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:09:55.510878 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505272 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:09:55.510878 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505275 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:09:55.510878 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505277 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:09:55.510878 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505280 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:09:55.510878 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505283 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:09:55.510878 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505285 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:09:55.510878 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505287 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:09:55.510878 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505290 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:09:55.510878 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505292 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:09:55.510878 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505295 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:09:55.510878 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505297 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:09:55.510878 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505300 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:09:55.510878 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505302 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:09:55.510878 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505304 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:09:55.510878 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505307 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:09:55.511403 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505309 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:09:55.511403 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505311 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:09:55.511403 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505314 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:09:55.511403 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505316 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:09:55.511403 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505319 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:09:55.511403 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505321 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:09:55.511403 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505324 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:09:55.511403 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505326 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:09:55.511403 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505328 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:09:55.511403 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505331 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:09:55.511403 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505333 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:09:55.511403 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505336 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:09:55.511403 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505338 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:09:55.511403 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505340 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:09:55.511403 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505343 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:09:55.511403 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505345 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:09:55.511403 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505348 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:09:55.511403 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505350 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:09:55.511403 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505353 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:09:55.511403 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505355 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:09:55.511894 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505357 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:09:55.511894 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505359 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:09:55.511894 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505362 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:09:55.511894 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505364 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:09:55.511894 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505367 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:09:55.511894 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505369 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:09:55.511894 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:55.505372 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:09:55.511894 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.505377 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 18:09:55.511894 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.505498 2565 server.go:962] "Client rotation is on, will bootstrap in background" Jul 14 18:09:55.511894 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.507810 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 14 18:09:55.511894 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.509351 2565 server.go:1019] "Starting client certificate rotation" Jul 14 18:09:55.511894 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.509457 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 18:09:55.511894 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.509494 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 18:09:55.512237 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.512123 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 18:09:55.513257 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.513238 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 18:09:55.521862 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.521708 2565 log.go:25] "Validated CRI v1 runtime API" Jul 14 18:09:55.526449 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.526432 2565 log.go:25] "Validated CRI v1 image API" Jul 14 18:09:55.527269 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.527255 2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 14 18:09:55.529819 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.529796 2565 fs.go:135] Filesystem UUIDs: map[4a9c4737-01e5-432a-9991-19a491627e54:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f1992c93-20e1-4a2a-b857-2c0071f49120:/dev/nvme0n1p4] Jul 14 18:09:55.529889 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.529817 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 14 18:09:55.535427 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.535323 2565 manager.go:216] Machine: {Timestamp:2026-07-14 18:09:55.533923359 +0000 UTC m=+0.371266608 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3055897 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2133a37232c13fd0f89d573a378577 SystemUUID:ec2133a3-7232-c13f-d0f8-9d573a378577 BootID:b05067e0-5511-45bc-907b-9c2287b69fca Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a3:76:f9:4c:11 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a3:76:f9:4c:11 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:89:08:b7:19:23 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 14 18:09:55.535427 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.535425 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 14 18:09:55.535537 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.535498 2565 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 14 18:09:55.538764 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.538739 2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 14 18:09:55.538900 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.538768 2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-25.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 14 18:09:55.539641 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.539631 2565 topology_manager.go:138] "Creating topology manager with none policy" Jul 14 18:09:55.539677 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.539644 2565 container_manager_linux.go:309] "Creating device plugin manager" Jul 14 18:09:55.539677 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.539652 2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 18:09:55.540483 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.540471 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 18:09:55.542005 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.541995 2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 14 18:09:55.546047 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.546033 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 14 18:09:55.546230 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.546218 2565 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 14 18:09:55.546820 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.546811 2565 kubelet.go:486] "Attempting to sync node with API server" Jul 14 18:09:55.546858 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.546824 2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 14 18:09:55.546858 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.546838 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 14 18:09:55.546858 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.546847 2565 kubelet.go:398] "Adding apiserver pod source" Jul 14 18:09:55.546858 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.546855 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 14 18:09:55.547249 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.547236 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 18:09:55.547309 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.547265 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 18:09:55.549168 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.549150 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 18:09:55.549949 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.549935 2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 14 18:09:55.552419 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.552406 2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 14 18:09:55.552467 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.552435 2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 14 18:09:55.552638 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.552628 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 14 18:09:55.552677 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.552643 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 14 18:09:55.552677 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.552649 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 14 18:09:55.552677 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.552655 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 14 18:09:55.552677 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.552663 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 14 18:09:55.552677 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.552669 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 14 18:09:55.553297 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.553287 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 14 18:09:55.553297 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.553297 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 14 18:09:55.553358 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.553303 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 14 18:09:55.553358 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.553309 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 14 18:09:55.553358 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.553318 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 14 18:09:55.553358 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.553326 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 14 18:09:55.554250 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.554238 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 14 18:09:55.554288 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.554252 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 14 18:09:55.557387 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.557373 2565 server.go:1268] "Started kubelet" Jul 14 18:09:55.557466 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.557440 2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 14 18:09:55.557552 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.557511 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 14 18:09:55.557651 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.557569 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 14 18:09:55.558384 ip-10-0-136-25 systemd[1]: Started Kubernetes Kubelet. Jul 14 18:09:55.560038 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.560020 2565 server.go:310] "Adding debug handlers to kubelet server" Jul 14 18:09:55.560334 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.560302 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-25.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 14 18:09:55.560447 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:55.560426 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-25.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 14 18:09:55.560511 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:55.560436 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 14 18:09:55.561965 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.561940 2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 14 18:09:55.564491 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:55.564459 2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 14 18:09:55.565511 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:55.564427 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-25.ec2.internal.18c23a6222cede3e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-25.ec2.internal,UID:ip-10-0-136-25.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-25.ec2.internal,},FirstTimestamp:2026-07-14 18:09:55.557326398 +0000 UTC m=+0.394669647,LastTimestamp:2026-07-14 18:09:55.557326398 +0000 UTC m=+0.394669647,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-25.ec2.internal,}" Jul 14 18:09:55.569267 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.569253 2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 14 18:09:55.569267 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.569262 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 14 18:09:55.569371 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:55.569324 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-25.ec2.internal\" not found" Jul 14 18:09:55.570028 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.570009 2565 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 14 18:09:55.570028 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.570030 2565 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 14 18:09:55.570201 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.570009 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 14 18:09:55.570201 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.570139 2565 reconstruct.go:97] "Volume reconstruction finished" Jul 14 18:09:55.570201 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.570148 2565 reconciler.go:29] "Reconciler: start to sync state" Jul 14 18:09:55.570340 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.570215 2565 factory.go:153] Registering CRI-O factory Jul 14 18:09:55.570340 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.570225 2565 factory.go:223] Registration of the crio container factory successfully Jul 14 18:09:55.570340 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.570269 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 14 18:09:55.570340 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.570277 2565 factory.go:55] Registering systemd factory Jul 14 18:09:55.570340 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.570283 2565 factory.go:223] Registration of the systemd container factory successfully Jul 14 18:09:55.570340 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.570297 2565 factory.go:103] Registering Raw factory Jul 14 18:09:55.570340 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.570306 2565 manager.go:1196] Started watching for new ooms in manager Jul 14 18:09:55.571406 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.571391 2565 manager.go:349] Starting recovery of all containers Jul 14 18:09:55.573510 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:55.573386 2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-25.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 14 18:09:55.573510 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:55.573461 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 14 18:09:55.582608 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.582345 2565 manager.go:354] Recovery completed Jul 14 18:09:55.584223 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:55.584181 2565 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 14 18:09:55.587797 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.587782 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:09:55.589611 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.589595 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:09:55.589683 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.589629 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:09:55.589683 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.589642 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:09:55.590216 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.590202 2565 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 14 18:09:55.590216 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.590213 2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 14 18:09:55.590352 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.590224 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:09:55.590352 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.590232 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 14 18:09:55.591300 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.591283 2565 policy_none.go:49] "None policy: Start" Jul 14 18:09:55.591353 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.591304 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 14 18:09:55.591353 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.591315 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 14 18:09:55.591934 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.591919 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:09:55.591986 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.591949 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:09:55.591986 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.591959 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:09:55.592310 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.592297 2565 policy_none.go:47] "Start" Jul 14 18:09:55.592377 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:55.592281 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-25.ec2.internal.18c23a6224bb8124 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-25.ec2.internal,UID:ip-10-0-136-25.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-25.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-25.ec2.internal,},FirstTimestamp:2026-07-14 18:09:55.589611812 +0000 UTC m=+0.426955062,LastTimestamp:2026-07-14 18:09:55.589611812 +0000 UTC m=+0.426955062,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-25.ec2.internal,}" Jul 14 18:09:55.602946 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:55.602634 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-25.ec2.internal.18c23a6224bbe156 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-25.ec2.internal,UID:ip-10-0-136-25.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-25.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-25.ec2.internal,},FirstTimestamp:2026-07-14 18:09:55.589636438 +0000 UTC m=+0.426979686,LastTimestamp:2026-07-14 18:09:55.589636438 +0000 UTC m=+0.426979686,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-25.ec2.internal,}" Jul 14 18:09:55.615167 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:55.615086 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-25.ec2.internal.18c23a6224bc08c5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-25.ec2.internal,UID:ip-10-0-136-25.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-25.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-25.ec2.internal,},FirstTimestamp:2026-07-14 18:09:55.589646533 +0000 UTC m=+0.426989781,LastTimestamp:2026-07-14 18:09:55.589646533 +0000 UTC m=+0.426989781,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-25.ec2.internal,}" Jul 14 18:09:55.615538 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.615520 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gmv6s" Jul 14 18:09:55.624772 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.624756 2565 manager.go:337] "Starting Device Plugin manager" Jul 14 18:09:55.624859 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:55.624794 2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 14 18:09:55.624859 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.624807 2565 server.go:87] "Starting device plugin registration server" Jul 14 18:09:55.624859 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:55.624754 2565 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-25.ec2.internal.18c23a6224bb8124\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-25.ec2.internal.18c23a6224bb8124 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-25.ec2.internal,UID:ip-10-0-136-25.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-25.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-25.ec2.internal,},FirstTimestamp:2026-07-14 18:09:55.589611812 +0000 UTC m=+0.426955062,LastTimestamp:2026-07-14 18:09:55.59193583 +0000 UTC m=+0.429279079,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-25.ec2.internal,}" Jul 14 18:09:55.625062 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.625050 2565 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 14 18:09:55.625127 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.625066 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 14 18:09:55.625179 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.625160 2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 14 18:09:55.625244 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.625223 2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 14 18:09:55.625244 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.625235 2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 14 18:09:55.625762 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:55.625744 2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 14 18:09:55.625833 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:55.625796 2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-25.ec2.internal\" not found" Jul 14 18:09:55.631240 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.631221 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gmv6s" Jul 14 18:09:55.644449 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:55.632281 2565 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-25.ec2.internal.18c23a6224bbe156\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-25.ec2.internal.18c23a6224bbe156 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-25.ec2.internal,UID:ip-10-0-136-25.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-25.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-25.ec2.internal,},FirstTimestamp:2026-07-14 18:09:55.589636438 +0000 UTC m=+0.426979686,LastTimestamp:2026-07-14 18:09:55.59195447 +0000 UTC m=+0.429297719,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-25.ec2.internal,}" Jul 14 18:09:55.682182 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.682152 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 14 18:09:55.683487 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.683469 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 14 18:09:55.683572 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.683511 2565 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 14 18:09:55.683572 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.683538 2565 kubelet.go:2448] "Starting kubelet main sync loop" Jul 14 18:09:55.683665 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:55.683575 2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 14 18:09:55.686312 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.686294 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:09:55.725850 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.725827 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:09:55.727052 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.727036 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:09:55.727123 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.727068 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:09:55.727123 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.727081 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:09:55.727123 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.727119 2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-25.ec2.internal" Jul 14 18:09:55.738706 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.738664 2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-25.ec2.internal" Jul 14 18:09:55.738706 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:55.738685 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-25.ec2.internal\": node \"ip-10-0-136-25.ec2.internal\" not found" Jul 14 18:09:55.758878 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:55.758859 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-25.ec2.internal\" not found" Jul 14 18:09:55.783862 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.783838 2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-25.ec2.internal"] Jul 14 18:09:55.783944 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.783895 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:09:55.785376 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.785363 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:09:55.785430 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.785390 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:09:55.785430 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.785399 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:09:55.786587 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.786575 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:09:55.786693 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.786680 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal" Jul 14 18:09:55.786733 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.786708 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:09:55.787281 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.787267 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:09:55.787373 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.787286 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:09:55.787373 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.787301 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:09:55.787373 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.787314 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:09:55.787373 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.787316 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:09:55.787373 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.787324 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:09:55.788358 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.788344 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-25.ec2.internal" Jul 14 18:09:55.788407 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.788377 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:09:55.789010 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.788992 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:09:55.789090 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.789027 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:09:55.789090 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.789041 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:09:55.817719 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:55.817696 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-25.ec2.internal\" not found" node="ip-10-0-136-25.ec2.internal" Jul 14 18:09:55.821982 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:55.821967 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-25.ec2.internal\" not found" node="ip-10-0-136-25.ec2.internal" Jul 14 18:09:55.859312 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:55.859285 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-25.ec2.internal\" not found" Jul 14 18:09:55.871452 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.871429 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c5ad4d84a227248b0267edbeaea1f57-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal\" (UID: \"5c5ad4d84a227248b0267edbeaea1f57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal" Jul 14 18:09:55.871536 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.871455 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/03b994a435c6ac42ecdd2d02de3d23bb-config\") pod \"kube-apiserver-proxy-ip-10-0-136-25.ec2.internal\" (UID: \"03b994a435c6ac42ecdd2d02de3d23bb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-25.ec2.internal" Jul 14 18:09:55.871536 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.871478 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5c5ad4d84a227248b0267edbeaea1f57-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal\" (UID: \"5c5ad4d84a227248b0267edbeaea1f57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal" Jul 14 18:09:55.959745 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:55.959725 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-25.ec2.internal\" not found" Jul 14 18:09:55.972082 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.972061 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/03b994a435c6ac42ecdd2d02de3d23bb-config\") pod \"kube-apiserver-proxy-ip-10-0-136-25.ec2.internal\" (UID: \"03b994a435c6ac42ecdd2d02de3d23bb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-25.ec2.internal" Jul 14 18:09:55.972149 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.972089 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5c5ad4d84a227248b0267edbeaea1f57-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal\" (UID: \"5c5ad4d84a227248b0267edbeaea1f57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal" Jul 14 18:09:55.972149 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.972130 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c5ad4d84a227248b0267edbeaea1f57-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal\" (UID: \"5c5ad4d84a227248b0267edbeaea1f57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal" Jul 14 18:09:55.972212 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.972163 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c5ad4d84a227248b0267edbeaea1f57-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal\" (UID: \"5c5ad4d84a227248b0267edbeaea1f57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal" Jul 14 18:09:55.972212 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.972164 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5c5ad4d84a227248b0267edbeaea1f57-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal\" (UID: \"5c5ad4d84a227248b0267edbeaea1f57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal" Jul 14 18:09:55.972212 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:55.972166 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/03b994a435c6ac42ecdd2d02de3d23bb-config\") pod \"kube-apiserver-proxy-ip-10-0-136-25.ec2.internal\" (UID: \"03b994a435c6ac42ecdd2d02de3d23bb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-25.ec2.internal" Jul 14 18:09:56.060483 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:56.060462 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-25.ec2.internal\" not found" Jul 14 18:09:56.120017 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:56.119986 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal" Jul 14 18:09:56.124899 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:56.124884 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-25.ec2.internal" Jul 14 18:09:56.161407 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:56.161377 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-25.ec2.internal\" not found" Jul 14 18:09:56.261958 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:56.261919 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-25.ec2.internal\" not found" Jul 14 18:09:56.362474 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:56.362398 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-25.ec2.internal\" not found" Jul 14 18:09:56.463137 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:56.463082 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-25.ec2.internal\" not found" Jul 14 18:09:56.508562 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:56.508534 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 14 18:09:56.509121 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:56.508674 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 14 18:09:56.540911 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:56.540809 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:09:56.563615 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:56.563576 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-25.ec2.internal\" not found" Jul 14 18:09:56.635574 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:56.635507 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 18:04:55 +0000 UTC" deadline="2028-03-23 17:13:59.60545526 +0000 UTC" Jul 14 18:09:56.635574 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:56.635540 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14831h4m2.969921069s" Jul 14 18:09:56.663439 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:56.663404 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c5ad4d84a227248b0267edbeaea1f57.slice/crio-2c8a50e4b5eec5211ff543cd82b315e008f213a43101b43b898727e03141dd25 WatchSource:0}: Error finding container 2c8a50e4b5eec5211ff543cd82b315e008f213a43101b43b898727e03141dd25: Status 404 returned error can't find the container with id 2c8a50e4b5eec5211ff543cd82b315e008f213a43101b43b898727e03141dd25 Jul 14 18:09:56.663756 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:56.663740 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b994a435c6ac42ecdd2d02de3d23bb.slice/crio-b853839e914219e61844a229a19998861be3dd10edf5043a9416a4728f2f5620 WatchSource:0}: Error finding container b853839e914219e61844a229a19998861be3dd10edf5043a9416a4728f2f5620: Status 404 returned error can't find the container with id b853839e914219e61844a229a19998861be3dd10edf5043a9416a4728f2f5620 Jul 14 18:09:56.663756 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:56.663749 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-25.ec2.internal\" not found" Jul 14 18:09:56.667869 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:56.667856 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 18:09:56.687111 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:56.687052 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-25.ec2.internal" event={"ID":"03b994a435c6ac42ecdd2d02de3d23bb","Type":"ContainerStarted","Data":"b853839e914219e61844a229a19998861be3dd10edf5043a9416a4728f2f5620"} Jul 14 18:09:56.688285 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:56.688262 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal" event={"ID":"5c5ad4d84a227248b0267edbeaea1f57","Type":"ContainerStarted","Data":"2c8a50e4b5eec5211ff543cd82b315e008f213a43101b43b898727e03141dd25"} Jul 14 18:09:56.764532 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:56.764498 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-25.ec2.internal\" not found" Jul 14 18:09:56.769652 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:56.769637 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:09:56.770725 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:56.770707 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:09:56.770818 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:56.770738 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:09:56.770818 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:56.770748 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:09:56.865588 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:56.865561 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-25.ec2.internal\" not found" Jul 14 18:09:56.873813 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:56.873796 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:09:56.966239 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:56.966178 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-25.ec2.internal\" not found" Jul 14 18:09:57.066952 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:57.066912 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-25.ec2.internal\" not found" Jul 14 18:09:57.149547 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.149526 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:09:57.170013 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.169991 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal" Jul 14 18:09:57.191740 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.191720 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 18:09:57.193043 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.193030 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-25.ec2.internal" Jul 14 18:09:57.207559 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.207544 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 18:09:57.549457 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.549424 2565 apiserver.go:52] "Watching apiserver" Jul 14 18:09:57.562519 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.562494 2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 14 18:09:57.562872 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.562850 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dj76","openshift-cluster-node-tuning-operator/tuned-nhrbt","openshift-image-registry/node-ca-qx9gz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal","openshift-multus/network-metrics-daemon-rbgtx","openshift-network-operator/iptables-alerter-lrhsn","kube-system/global-pull-secret-syncer-6czgz","kube-system/konnectivity-agent-sgdnj","kube-system/kube-apiserver-proxy-ip-10-0-136-25.ec2.internal","openshift-multus/multus-additional-cni-plugins-dmt89","openshift-multus/multus-h7jw8","openshift-network-diagnostics/network-check-target-x92qm","openshift-ovn-kubernetes/ovnkube-node-f99bz"] Jul 14 18:09:57.566933 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.566917 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dmt89" Jul 14 18:09:57.569281 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.569262 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.570503 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.570475 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 14 18:09:57.570503 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.570487 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 14 18:09:57.570653 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.570561 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 14 18:09:57.570653 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.570630 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 14 18:09:57.570653 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.570647 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 14 18:09:57.570828 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.570780 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h4trv\"" Jul 14 18:09:57.570828 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.570787 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 14 18:09:57.571382 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.571365 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:09:57.571461 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:57.571429 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6czgz" podUID="015d11c8-2009-4c76-815a-c2d87c42e7b5" Jul 14 18:09:57.571905 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.571891 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 14 18:09:57.571996 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.571983 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-z58pp\"" Jul 14 18:09:57.573914 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.573706 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:09:57.573914 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:57.573774 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbgtx" podUID="a1fc5b88-ec77-4f28-83ff-82725cf2a5e1" Jul 14 18:09:57.576272 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.576250 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lrhsn" Jul 14 18:09:57.578600 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.578583 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dj76" Jul 14 18:09:57.579359 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.579337 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h2xqc\"" Jul 14 18:09:57.579359 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.579344 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 14 18:09:57.579490 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.579395 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 14 18:09:57.579490 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.579460 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 14 18:09:57.580929 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.580909 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sgdnj" Jul 14 18:09:57.581542 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.581525 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 14 18:09:57.581810 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.581787 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 14 18:09:57.581914 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.581787 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 14 18:09:57.582173 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.582153 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5gnt5\"" Jul 14 18:09:57.582792 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.582776 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-multus-cni-dir\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.582859 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.582803 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-multus-socket-dir-parent\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.582859 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.582821 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-host-run-multus-certs\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.582859 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.582844 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/015d11c8-2009-4c76-815a-c2d87c42e7b5-original-pull-secret\") pod \"global-pull-secret-syncer-6czgz\" (UID: \"015d11c8-2009-4c76-815a-c2d87c42e7b5\") " pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:09:57.582964 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.582877 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-host-var-lib-cni-bin\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.582964 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.582912 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dmt89\" (UID: \"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5\") " pod="openshift-multus/multus-additional-cni-plugins-dmt89" Jul 14 18:09:57.582964 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.582943 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dmt89\" (UID: \"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5\") " pod="openshift-multus/multus-additional-cni-plugins-dmt89" Jul 14 18:09:57.582964 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.582960 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-etc-kubernetes\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.583104 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.582986 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/015d11c8-2009-4c76-815a-c2d87c42e7b5-dbus\") pod \"global-pull-secret-syncer-6czgz\" (UID: \"015d11c8-2009-4c76-815a-c2d87c42e7b5\") " pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:09:57.583104 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.583009 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-os-release\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.583104 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.583025 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-host-run-k8s-cni-cncf-io\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.583104 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.583041 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-host-run-netns\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.583104 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.583056 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-hostroot\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.583104 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.583079 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dcdcb581-e08c-4d47-90ae-bc03b850964d-multus-daemon-config\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.583336 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.583101 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4fbc\" (UniqueName: \"kubernetes.io/projected/dcdcb581-e08c-4d47-90ae-bc03b850964d-kube-api-access-c4fbc\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.583336 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.583167 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-cnibin\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.583336 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.583201 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5-os-release\") pod \"multus-additional-cni-plugins-dmt89\" (UID: \"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5\") " pod="openshift-multus/multus-additional-cni-plugins-dmt89" Jul 14 18:09:57.583336 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.583231 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5-cni-binary-copy\") pod \"multus-additional-cni-plugins-dmt89\" (UID: \"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5\") " pod="openshift-multus/multus-additional-cni-plugins-dmt89" Jul 14 18:09:57.583336 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.583255 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5-system-cni-dir\") pod \"multus-additional-cni-plugins-dmt89\" (UID: \"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5\") " pod="openshift-multus/multus-additional-cni-plugins-dmt89" Jul 14 18:09:57.583336 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.583281 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5-cnibin\") pod \"multus-additional-cni-plugins-dmt89\" (UID: \"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5\") " pod="openshift-multus/multus-additional-cni-plugins-dmt89" Jul 14 18:09:57.583336 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.583305 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdtgl\" (UniqueName: \"kubernetes.io/projected/42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5-kube-api-access-pdtgl\") pod \"multus-additional-cni-plugins-dmt89\" (UID: \"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5\") " pod="openshift-multus/multus-additional-cni-plugins-dmt89" Jul 14 18:09:57.583336 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.583318 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qx9gz" Jul 14 18:09:57.583336 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.583331 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-host-var-lib-kubelet\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.583772 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.583365 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dmt89\" (UID: \"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5\") " pod="openshift-multus/multus-additional-cni-plugins-dmt89" Jul 14 18:09:57.583772 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.583388 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/015d11c8-2009-4c76-815a-c2d87c42e7b5-kubelet-config\") pod \"global-pull-secret-syncer-6czgz\" (UID: \"015d11c8-2009-4c76-815a-c2d87c42e7b5\") " pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:09:57.583772 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.583434 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-system-cni-dir\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.583772 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.583496 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dcdcb581-e08c-4d47-90ae-bc03b850964d-cni-binary-copy\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.583772 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.583526 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-host-var-lib-cni-multus\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.583772 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.583561 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-multus-conf-dir\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.583772 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.583769 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wgqnn\"" Jul 14 18:09:57.583772 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.583776 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 14 18:09:57.584043 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.583782 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 14 18:09:57.585730 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.585715 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.588046 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.588031 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x92qm" Jul 14 18:09:57.588159 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:57.588095 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x92qm" podUID="65462b89-c7cc-4843-a6e9-baa2aac800af" Jul 14 18:09:57.588685 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.588668 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 14 18:09:57.588765 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.588669 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s2k55\"" Jul 14 18:09:57.589096 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.589082 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 14 18:09:57.589326 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.589311 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hqzfk\"" Jul 14 18:09:57.590213 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.590191 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 14 18:09:57.590299 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.590231 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 14 18:09:57.590299 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.590248 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 14 18:09:57.590497 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.590479 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.596037 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.596016 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 14 18:09:57.596115 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.596045 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 14 18:09:57.596493 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.596478 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 14 18:09:57.596733 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.596717 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 14 18:09:57.598139 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.598063 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 14 18:09:57.598573 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.598551 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 18:09:57.599008 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.598988 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 14 18:09:57.600783 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.599290 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-57hlz\"" Jul 14 18:09:57.624497 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.624478 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hjfsm" Jul 14 18:09:57.626638 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.626618 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:09:57.636456 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.636439 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hjfsm" Jul 14 18:09:57.671373 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.671348 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 14 18:09:57.683902 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.683875 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-var-lib-openvswitch\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.683996 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.683914 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-etc-kubernetes\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.683996 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.683934 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-lib-modules\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.683996 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.683955 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4fa88c56-d6f8-49af-90ec-11f33b911e81-device-dir\") pod \"aws-ebs-csi-driver-node-8dj76\" (UID: \"4fa88c56-d6f8-49af-90ec-11f33b911e81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dj76" Jul 14 18:09:57.684178 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.683994 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.684178 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684025 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-ovn-node-metrics-cert\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.684178 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684000 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-etc-kubernetes\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.684178 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684048 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-etc-sysctl-d\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.684178 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684066 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ff7a5a9c-2fae-4a88-90bd-f50314fda904-iptables-alerter-script\") pod \"iptables-alerter-lrhsn\" (UID: \"ff7a5a9c-2fae-4a88-90bd-f50314fda904\") " pod="openshift-network-operator/iptables-alerter-lrhsn" Jul 14 18:09:57.684178 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684094 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-host-kubelet\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.684178 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684143 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-run-ovn\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.684178 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684167 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-host-cni-bin\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.684491 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684208 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-host-cni-netd\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.684491 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684248 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-os-release\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.684491 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684264 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-host\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.684491 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684299 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-ovnkube-script-lib\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.684491 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684325 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/015d11c8-2009-4c76-815a-c2d87c42e7b5-kubelet-config\") pod \"global-pull-secret-syncer-6czgz\" (UID: \"015d11c8-2009-4c76-815a-c2d87c42e7b5\") " pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:09:57.684491 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684349 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-etc-kubernetes\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.684491 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684357 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-os-release\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.684491 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684372 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d953694d-140b-430a-aa16-6c0d82c8daa6-etc-tuned\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.684491 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684415 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-host-run-k8s-cni-cncf-io\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.684491 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684418 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/015d11c8-2009-4c76-815a-c2d87c42e7b5-kubelet-config\") pod \"global-pull-secret-syncer-6czgz\" (UID: \"015d11c8-2009-4c76-815a-c2d87c42e7b5\") " pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:09:57.684491 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684435 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-host-run-netns\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.684491 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684457 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-hostroot\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.684491 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684469 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-host-run-k8s-cni-cncf-io\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.684491 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684476 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dcdcb581-e08c-4d47-90ae-bc03b850964d-multus-daemon-config\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.685032 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684500 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-cnibin\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.685032 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684509 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-host-run-netns\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.685032 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684512 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-hostroot\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.685032 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684562 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-cnibin\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.685032 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684591 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5-system-cni-dir\") pod \"multus-additional-cni-plugins-dmt89\" (UID: \"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5\") " pod="openshift-multus/multus-additional-cni-plugins-dmt89" Jul 14 18:09:57.685032 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684616 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5-cnibin\") pod \"multus-additional-cni-plugins-dmt89\" (UID: \"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5\") " pod="openshift-multus/multus-additional-cni-plugins-dmt89" Jul 14 18:09:57.685032 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684659 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdtgl\" (UniqueName: \"kubernetes.io/projected/42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5-kube-api-access-pdtgl\") pod \"multus-additional-cni-plugins-dmt89\" (UID: \"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5\") " pod="openshift-multus/multus-additional-cni-plugins-dmt89" Jul 14 18:09:57.685032 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684664 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5-system-cni-dir\") pod \"multus-additional-cni-plugins-dmt89\" (UID: \"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5\") " pod="openshift-multus/multus-additional-cni-plugins-dmt89" Jul 14 18:09:57.685032 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684685 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-host-var-lib-kubelet\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.685032 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684710 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-run\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.685032 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684714 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5-cnibin\") pod \"multus-additional-cni-plugins-dmt89\" (UID: \"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5\") " pod="openshift-multus/multus-additional-cni-plugins-dmt89" Jul 14 18:09:57.685032 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684736 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff7a5a9c-2fae-4a88-90bd-f50314fda904-host-slash\") pod \"iptables-alerter-lrhsn\" (UID: \"ff7a5a9c-2fae-4a88-90bd-f50314fda904\") " pod="openshift-network-operator/iptables-alerter-lrhsn" Jul 14 18:09:57.685032 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684756 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-host-var-lib-kubelet\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.685032 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684775 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4fa88c56-d6f8-49af-90ec-11f33b911e81-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8dj76\" (UID: \"4fa88c56-d6f8-49af-90ec-11f33b911e81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dj76" Jul 14 18:09:57.685032 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684799 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-host-run-ovn-kubernetes\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.685032 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684823 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-system-cni-dir\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.685032 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684848 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-host-var-lib-cni-multus\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.685674 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684877 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/87ca2cc2-58d4-4899-8bc1-2b5bd93ebe6f-serviceca\") pod \"node-ca-qx9gz\" (UID: \"87ca2cc2-58d4-4899-8bc1-2b5bd93ebe6f\") " pod="openshift-image-registry/node-ca-qx9gz" Jul 14 18:09:57.685674 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684899 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-host-slash\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.685674 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684921 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-ovnkube-config\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.685674 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684955 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-multus-cni-dir\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.685674 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.684979 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/015d11c8-2009-4c76-815a-c2d87c42e7b5-original-pull-secret\") pod \"global-pull-secret-syncer-6czgz\" (UID: \"015d11c8-2009-4c76-815a-c2d87c42e7b5\") " pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:09:57.685674 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.685054 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-var-lib-kubelet\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.685674 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.685074 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-system-cni-dir\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.685674 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.685077 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dcdcb581-e08c-4d47-90ae-bc03b850964d-multus-daemon-config\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.685674 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.685130 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-host-var-lib-cni-multus\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.685674 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.685082 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhhw6\" (UniqueName: \"kubernetes.io/projected/65462b89-c7cc-4843-a6e9-baa2aac800af-kube-api-access-vhhw6\") pod \"network-check-target-x92qm\" (UID: \"65462b89-c7cc-4843-a6e9-baa2aac800af\") " pod="openshift-network-diagnostics/network-check-target-x92qm" Jul 14 18:09:57.685674 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.685180 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e3d8f9e9-f69e-4a3d-b688-ff64a5ec495e-agent-certs\") pod \"konnectivity-agent-sgdnj\" (UID: \"e3d8f9e9-f69e-4a3d-b688-ff64a5ec495e\") " pod="kube-system/konnectivity-agent-sgdnj" Jul 14 18:09:57.685674 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.685207 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-host-var-lib-cni-bin\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.685674 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.685234 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dmt89\" (UID: \"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5\") " pod="openshift-multus/multus-additional-cni-plugins-dmt89" Jul 14 18:09:57.685674 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:57.685247 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:09:57.685674 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.685259 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4l4p\" (UniqueName: \"kubernetes.io/projected/a1fc5b88-ec77-4f28-83ff-82725cf2a5e1-kube-api-access-d4l4p\") pod \"network-metrics-daemon-rbgtx\" (UID: \"a1fc5b88-ec77-4f28-83ff-82725cf2a5e1\") " pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:09:57.685674 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.685287 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-run-systemd\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.685674 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:57.685320 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/015d11c8-2009-4c76-815a-c2d87c42e7b5-original-pull-secret podName:015d11c8-2009-4c76-815a-c2d87c42e7b5 nodeName:}" failed. No retries permitted until 2026-07-14 18:09:58.18528552 +0000 UTC m=+3.022628776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/015d11c8-2009-4c76-815a-c2d87c42e7b5-original-pull-secret") pod "global-pull-secret-syncer-6czgz" (UID: "015d11c8-2009-4c76-815a-c2d87c42e7b5") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:09:57.686519 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.685339 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-host-var-lib-cni-bin\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.686519 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.685344 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-multus-cni-dir\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.686519 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.685391 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/015d11c8-2009-4c76-815a-c2d87c42e7b5-dbus\") pod \"global-pull-secret-syncer-6czgz\" (UID: \"015d11c8-2009-4c76-815a-c2d87c42e7b5\") " pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:09:57.686519 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.685415 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-etc-sysctl-conf\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.686519 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.685440 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-host-run-netns\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.686519 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.685464 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drkh8\" (UniqueName: \"kubernetes.io/projected/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-kube-api-access-drkh8\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.686519 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.685488 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-etc-modprobe-d\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.686519 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.685504 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dmt89\" (UID: \"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5\") " pod="openshift-multus/multus-additional-cni-plugins-dmt89" Jul 14 18:09:57.686519 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.685510 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-systemd-units\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.686519 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.685542 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-run-openvswitch\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.686519 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.685563 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-log-socket\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.686519 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.685592 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-env-overrides\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.686519 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.685620 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dmt89\" (UID: \"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5\") " pod="openshift-multus/multus-additional-cni-plugins-dmt89" Jul 14 18:09:57.686519 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.685610 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/015d11c8-2009-4c76-815a-c2d87c42e7b5-dbus\") pod \"global-pull-secret-syncer-6czgz\" (UID: \"015d11c8-2009-4c76-815a-c2d87c42e7b5\") " pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:09:57.686519 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.685647 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1fc5b88-ec77-4f28-83ff-82725cf2a5e1-metrics-certs\") pod \"network-metrics-daemon-rbgtx\" (UID: \"a1fc5b88-ec77-4f28-83ff-82725cf2a5e1\") " pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:09:57.686519 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.685672 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s2l8\" (UniqueName: \"kubernetes.io/projected/ff7a5a9c-2fae-4a88-90bd-f50314fda904-kube-api-access-9s2l8\") pod \"iptables-alerter-lrhsn\" (UID: \"ff7a5a9c-2fae-4a88-90bd-f50314fda904\") " pod="openshift-network-operator/iptables-alerter-lrhsn" Jul 14 18:09:57.686519 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.685696 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4fa88c56-d6f8-49af-90ec-11f33b911e81-socket-dir\") pod \"aws-ebs-csi-driver-node-8dj76\" (UID: \"4fa88c56-d6f8-49af-90ec-11f33b911e81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dj76" Jul 14 18:09:57.687383 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.685750 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4fbc\" (UniqueName: \"kubernetes.io/projected/dcdcb581-e08c-4d47-90ae-bc03b850964d-kube-api-access-c4fbc\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.687383 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.685805 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5-os-release\") pod \"multus-additional-cni-plugins-dmt89\" (UID: \"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5\") " pod="openshift-multus/multus-additional-cni-plugins-dmt89" Jul 14 18:09:57.687383 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.685925 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5-cni-binary-copy\") pod \"multus-additional-cni-plugins-dmt89\" (UID: \"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5\") " pod="openshift-multus/multus-additional-cni-plugins-dmt89" Jul 14 18:09:57.687383 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.685964 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4fa88c56-d6f8-49af-90ec-11f33b911e81-etc-selinux\") pod \"aws-ebs-csi-driver-node-8dj76\" (UID: \"4fa88c56-d6f8-49af-90ec-11f33b911e81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dj76" Jul 14 18:09:57.687383 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.686015 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-etc-sysconfig\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.687383 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.686047 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p48x8\" (UniqueName: \"kubernetes.io/projected/d953694d-140b-430a-aa16-6c0d82c8daa6-kube-api-access-p48x8\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.687383 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.686075 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-etc-openvswitch\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.687383 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.686093 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5-os-release\") pod \"multus-additional-cni-plugins-dmt89\" (UID: \"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5\") " pod="openshift-multus/multus-additional-cni-plugins-dmt89" Jul 14 18:09:57.687383 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.686100 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-node-log\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.687383 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.686153 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dmt89\" (UID: \"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5\") " pod="openshift-multus/multus-additional-cni-plugins-dmt89" Jul 14 18:09:57.687383 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.686200 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dcdcb581-e08c-4d47-90ae-bc03b850964d-cni-binary-copy\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.687383 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.686231 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-multus-conf-dir\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.687383 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.686264 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-multus-socket-dir-parent\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.687383 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.686265 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-multus-conf-dir\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.687383 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.686317 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-multus-socket-dir-parent\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.687383 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.686370 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-host-run-multus-certs\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.687383 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.686407 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfhcd\" (UniqueName: \"kubernetes.io/projected/87ca2cc2-58d4-4899-8bc1-2b5bd93ebe6f-kube-api-access-mfhcd\") pod \"node-ca-qx9gz\" (UID: \"87ca2cc2-58d4-4899-8bc1-2b5bd93ebe6f\") " pod="openshift-image-registry/node-ca-qx9gz" Jul 14 18:09:57.687954 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.686432 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dcdcb581-e08c-4d47-90ae-bc03b850964d-host-run-multus-certs\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.687954 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.686485 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-sys\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.687954 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.686513 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4fa88c56-d6f8-49af-90ec-11f33b911e81-registration-dir\") pod \"aws-ebs-csi-driver-node-8dj76\" (UID: \"4fa88c56-d6f8-49af-90ec-11f33b911e81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dj76" Jul 14 18:09:57.687954 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.686586 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twzxg\" (UniqueName: \"kubernetes.io/projected/4fa88c56-d6f8-49af-90ec-11f33b911e81-kube-api-access-twzxg\") pod \"aws-ebs-csi-driver-node-8dj76\" (UID: \"4fa88c56-d6f8-49af-90ec-11f33b911e81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dj76" Jul 14 18:09:57.687954 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.686590 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5-cni-binary-copy\") pod \"multus-additional-cni-plugins-dmt89\" (UID: \"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5\") " pod="openshift-multus/multus-additional-cni-plugins-dmt89" Jul 14 18:09:57.687954 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.686613 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e3d8f9e9-f69e-4a3d-b688-ff64a5ec495e-konnectivity-ca\") pod \"konnectivity-agent-sgdnj\" (UID: \"e3d8f9e9-f69e-4a3d-b688-ff64a5ec495e\") " pod="kube-system/konnectivity-agent-sgdnj" Jul 14 18:09:57.687954 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.686659 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dmt89\" (UID: \"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5\") " pod="openshift-multus/multus-additional-cni-plugins-dmt89" Jul 14 18:09:57.687954 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.686679 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/87ca2cc2-58d4-4899-8bc1-2b5bd93ebe6f-host\") pod \"node-ca-qx9gz\" (UID: \"87ca2cc2-58d4-4899-8bc1-2b5bd93ebe6f\") " pod="openshift-image-registry/node-ca-qx9gz" Jul 14 18:09:57.687954 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.686703 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-etc-systemd\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.687954 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.686726 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d953694d-140b-430a-aa16-6c0d82c8daa6-tmp\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.687954 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.686752 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4fa88c56-d6f8-49af-90ec-11f33b911e81-sys-fs\") pod \"aws-ebs-csi-driver-node-8dj76\" (UID: \"4fa88c56-d6f8-49af-90ec-11f33b911e81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dj76" Jul 14 18:09:57.687954 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.686830 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dcdcb581-e08c-4d47-90ae-bc03b850964d-cni-binary-copy\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.687954 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.687155 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dmt89\" (UID: \"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5\") " pod="openshift-multus/multus-additional-cni-plugins-dmt89" Jul 14 18:09:57.695076 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.695052 2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 14 18:09:57.698754 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.698731 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4fbc\" (UniqueName: \"kubernetes.io/projected/dcdcb581-e08c-4d47-90ae-bc03b850964d-kube-api-access-c4fbc\") pod \"multus-h7jw8\" (UID: \"dcdcb581-e08c-4d47-90ae-bc03b850964d\") " pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.698878 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.698791 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdtgl\" (UniqueName: \"kubernetes.io/projected/42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5-kube-api-access-pdtgl\") pod \"multus-additional-cni-plugins-dmt89\" (UID: \"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5\") " pod="openshift-multus/multus-additional-cni-plugins-dmt89" Jul 14 18:09:57.787422 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.787384 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4fa88c56-d6f8-49af-90ec-11f33b911e81-etc-selinux\") pod \"aws-ebs-csi-driver-node-8dj76\" (UID: \"4fa88c56-d6f8-49af-90ec-11f33b911e81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dj76" Jul 14 18:09:57.787422 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.787426 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-etc-sysconfig\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.787654 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.787449 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p48x8\" (UniqueName: \"kubernetes.io/projected/d953694d-140b-430a-aa16-6c0d82c8daa6-kube-api-access-p48x8\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.787654 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.787525 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-etc-openvswitch\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.787654 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.787542 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-etc-sysconfig\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.787654 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.787564 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4fa88c56-d6f8-49af-90ec-11f33b911e81-etc-selinux\") pod \"aws-ebs-csi-driver-node-8dj76\" (UID: \"4fa88c56-d6f8-49af-90ec-11f33b911e81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dj76" Jul 14 18:09:57.787654 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.787565 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-node-log\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.787654 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.787602 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-node-log\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.787654 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.787596 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-etc-openvswitch\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.787654 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.787608 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfhcd\" (UniqueName: \"kubernetes.io/projected/87ca2cc2-58d4-4899-8bc1-2b5bd93ebe6f-kube-api-access-mfhcd\") pod \"node-ca-qx9gz\" (UID: \"87ca2cc2-58d4-4899-8bc1-2b5bd93ebe6f\") " pod="openshift-image-registry/node-ca-qx9gz" Jul 14 18:09:57.787654 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.787643 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-sys\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.788062 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.787671 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4fa88c56-d6f8-49af-90ec-11f33b911e81-registration-dir\") pod \"aws-ebs-csi-driver-node-8dj76\" (UID: \"4fa88c56-d6f8-49af-90ec-11f33b911e81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dj76" Jul 14 18:09:57.788062 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.787700 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twzxg\" (UniqueName: \"kubernetes.io/projected/4fa88c56-d6f8-49af-90ec-11f33b911e81-kube-api-access-twzxg\") pod \"aws-ebs-csi-driver-node-8dj76\" (UID: \"4fa88c56-d6f8-49af-90ec-11f33b911e81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dj76" Jul 14 18:09:57.788062 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.787728 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e3d8f9e9-f69e-4a3d-b688-ff64a5ec495e-konnectivity-ca\") pod \"konnectivity-agent-sgdnj\" (UID: \"e3d8f9e9-f69e-4a3d-b688-ff64a5ec495e\") " pod="kube-system/konnectivity-agent-sgdnj" Jul 14 18:09:57.788062 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.787750 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-sys\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.788062 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.787754 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/87ca2cc2-58d4-4899-8bc1-2b5bd93ebe6f-host\") pod \"node-ca-qx9gz\" (UID: \"87ca2cc2-58d4-4899-8bc1-2b5bd93ebe6f\") " pod="openshift-image-registry/node-ca-qx9gz" Jul 14 18:09:57.788062 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.787769 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4fa88c56-d6f8-49af-90ec-11f33b911e81-registration-dir\") pod \"aws-ebs-csi-driver-node-8dj76\" (UID: \"4fa88c56-d6f8-49af-90ec-11f33b911e81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dj76" Jul 14 18:09:57.788062 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.787788 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/87ca2cc2-58d4-4899-8bc1-2b5bd93ebe6f-host\") pod \"node-ca-qx9gz\" (UID: \"87ca2cc2-58d4-4899-8bc1-2b5bd93ebe6f\") " pod="openshift-image-registry/node-ca-qx9gz" Jul 14 18:09:57.788062 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.787792 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-etc-systemd\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.788062 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.787821 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d953694d-140b-430a-aa16-6c0d82c8daa6-tmp\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.788062 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.787848 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-etc-systemd\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.788062 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.787905 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4fa88c56-d6f8-49af-90ec-11f33b911e81-sys-fs\") pod \"aws-ebs-csi-driver-node-8dj76\" (UID: \"4fa88c56-d6f8-49af-90ec-11f33b911e81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dj76" Jul 14 18:09:57.788062 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.787848 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4fa88c56-d6f8-49af-90ec-11f33b911e81-sys-fs\") pod \"aws-ebs-csi-driver-node-8dj76\" (UID: \"4fa88c56-d6f8-49af-90ec-11f33b911e81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dj76" Jul 14 18:09:57.788062 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.787947 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-var-lib-openvswitch\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.788062 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.787994 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-lib-modules\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.788062 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788021 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4fa88c56-d6f8-49af-90ec-11f33b911e81-device-dir\") pod \"aws-ebs-csi-driver-node-8dj76\" (UID: \"4fa88c56-d6f8-49af-90ec-11f33b911e81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dj76" Jul 14 18:09:57.788062 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788048 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.788062 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788075 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-ovn-node-metrics-cert\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.788889 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788099 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-etc-sysctl-d\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.788889 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788124 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ff7a5a9c-2fae-4a88-90bd-f50314fda904-iptables-alerter-script\") pod \"iptables-alerter-lrhsn\" (UID: \"ff7a5a9c-2fae-4a88-90bd-f50314fda904\") " pod="openshift-network-operator/iptables-alerter-lrhsn" Jul 14 18:09:57.788889 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788166 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-host-kubelet\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.788889 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788189 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4fa88c56-d6f8-49af-90ec-11f33b911e81-device-dir\") pod \"aws-ebs-csi-driver-node-8dj76\" (UID: \"4fa88c56-d6f8-49af-90ec-11f33b911e81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dj76" Jul 14 18:09:57.788889 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788191 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-run-ovn\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.788889 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788238 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-host-cni-bin\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.788889 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788241 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-run-ovn\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.788889 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788260 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-host-cni-netd\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.788889 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788284 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-host\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.788889 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788305 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-ovnkube-script-lib\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.788889 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788298 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.788889 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788332 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-etc-kubernetes\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.788889 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788354 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-etc-sysctl-d\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.788889 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788370 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d953694d-140b-430a-aa16-6c0d82c8daa6-etc-tuned\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.788889 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788382 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-etc-kubernetes\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.788889 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788413 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-run\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.788889 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788440 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff7a5a9c-2fae-4a88-90bd-f50314fda904-host-slash\") pod \"iptables-alerter-lrhsn\" (UID: \"ff7a5a9c-2fae-4a88-90bd-f50314fda904\") " pod="openshift-network-operator/iptables-alerter-lrhsn" Jul 14 18:09:57.789672 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788457 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e3d8f9e9-f69e-4a3d-b688-ff64a5ec495e-konnectivity-ca\") pod \"konnectivity-agent-sgdnj\" (UID: \"e3d8f9e9-f69e-4a3d-b688-ff64a5ec495e\") " pod="kube-system/konnectivity-agent-sgdnj" Jul 14 18:09:57.789672 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788466 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4fa88c56-d6f8-49af-90ec-11f33b911e81-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8dj76\" (UID: \"4fa88c56-d6f8-49af-90ec-11f33b911e81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dj76" Jul 14 18:09:57.789672 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788493 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-host-run-ovn-kubernetes\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.789672 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788513 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-run\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.789672 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788518 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-lib-modules\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.789672 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788520 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/87ca2cc2-58d4-4899-8bc1-2b5bd93ebe6f-serviceca\") pod \"node-ca-qx9gz\" (UID: \"87ca2cc2-58d4-4899-8bc1-2b5bd93ebe6f\") " pod="openshift-image-registry/node-ca-qx9gz" Jul 14 18:09:57.789672 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788562 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-host-slash\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.789672 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788589 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-ovnkube-config\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.789672 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788630 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-var-lib-kubelet\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.789672 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788657 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhhw6\" (UniqueName: \"kubernetes.io/projected/65462b89-c7cc-4843-a6e9-baa2aac800af-kube-api-access-vhhw6\") pod \"network-check-target-x92qm\" (UID: \"65462b89-c7cc-4843-a6e9-baa2aac800af\") " pod="openshift-network-diagnostics/network-check-target-x92qm" Jul 14 18:09:57.789672 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788681 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e3d8f9e9-f69e-4a3d-b688-ff64a5ec495e-agent-certs\") pod \"konnectivity-agent-sgdnj\" (UID: \"e3d8f9e9-f69e-4a3d-b688-ff64a5ec495e\") " pod="kube-system/konnectivity-agent-sgdnj" Jul 14 18:09:57.789672 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788714 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4l4p\" (UniqueName: \"kubernetes.io/projected/a1fc5b88-ec77-4f28-83ff-82725cf2a5e1-kube-api-access-d4l4p\") pod \"network-metrics-daemon-rbgtx\" (UID: \"a1fc5b88-ec77-4f28-83ff-82725cf2a5e1\") " pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:09:57.789672 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788746 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-run-systemd\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.789672 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788771 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-etc-sysctl-conf\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.789672 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788796 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-host-run-netns\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.789672 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788821 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drkh8\" (UniqueName: \"kubernetes.io/projected/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-kube-api-access-drkh8\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.789672 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788844 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-etc-modprobe-d\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.790313 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788877 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-systemd-units\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.790313 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788895 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-host-run-ovn-kubernetes\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.790313 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788947 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-host-cni-bin\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.790313 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788949 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-run-openvswitch\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.790313 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788901 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-run-openvswitch\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.790313 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.788995 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-log-socket\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.790313 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.789036 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-env-overrides\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.790313 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.789067 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1fc5b88-ec77-4f28-83ff-82725cf2a5e1-metrics-certs\") pod \"network-metrics-daemon-rbgtx\" (UID: \"a1fc5b88-ec77-4f28-83ff-82725cf2a5e1\") " pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:09:57.790313 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.789039 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-host-cni-netd\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.790313 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.789096 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s2l8\" (UniqueName: \"kubernetes.io/projected/ff7a5a9c-2fae-4a88-90bd-f50314fda904-kube-api-access-9s2l8\") pod \"iptables-alerter-lrhsn\" (UID: \"ff7a5a9c-2fae-4a88-90bd-f50314fda904\") " pod="openshift-network-operator/iptables-alerter-lrhsn" Jul 14 18:09:57.790313 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.789222 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4fa88c56-d6f8-49af-90ec-11f33b911e81-socket-dir\") pod \"aws-ebs-csi-driver-node-8dj76\" (UID: \"4fa88c56-d6f8-49af-90ec-11f33b911e81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dj76" Jul 14 18:09:57.790313 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.789388 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4fa88c56-d6f8-49af-90ec-11f33b911e81-socket-dir\") pod \"aws-ebs-csi-driver-node-8dj76\" (UID: \"4fa88c56-d6f8-49af-90ec-11f33b911e81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dj76" Jul 14 18:09:57.790313 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.789414 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff7a5a9c-2fae-4a88-90bd-f50314fda904-host-slash\") pod \"iptables-alerter-lrhsn\" (UID: \"ff7a5a9c-2fae-4a88-90bd-f50314fda904\") " pod="openshift-network-operator/iptables-alerter-lrhsn" Jul 14 18:09:57.790313 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.789417 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4fa88c56-d6f8-49af-90ec-11f33b911e81-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8dj76\" (UID: \"4fa88c56-d6f8-49af-90ec-11f33b911e81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dj76" Jul 14 18:09:57.790313 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.789456 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-host\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.790313 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.789473 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-var-lib-openvswitch\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.790313 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.789517 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-log-socket\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.791105 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.789068 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-host-kubelet\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.791105 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:57.789618 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:09:57.791105 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:57.789680 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1fc5b88-ec77-4f28-83ff-82725cf2a5e1-metrics-certs podName:a1fc5b88-ec77-4f28-83ff-82725cf2a5e1 nodeName:}" failed. No retries permitted until 2026-07-14 18:09:58.289660922 +0000 UTC m=+3.127004176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1fc5b88-ec77-4f28-83ff-82725cf2a5e1-metrics-certs") pod "network-metrics-daemon-rbgtx" (UID: "a1fc5b88-ec77-4f28-83ff-82725cf2a5e1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:09:57.791105 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.789680 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/87ca2cc2-58d4-4899-8bc1-2b5bd93ebe6f-serviceca\") pod \"node-ca-qx9gz\" (UID: \"87ca2cc2-58d4-4899-8bc1-2b5bd93ebe6f\") " pod="openshift-image-registry/node-ca-qx9gz" Jul 14 18:09:57.791105 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.789740 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-host-run-netns\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.791105 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.789782 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-run-systemd\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.791105 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.789841 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-host-slash\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.791105 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.789912 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-etc-sysctl-conf\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.791105 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.790079 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-ovnkube-script-lib\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.791105 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.790088 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ff7a5a9c-2fae-4a88-90bd-f50314fda904-iptables-alerter-script\") pod \"iptables-alerter-lrhsn\" (UID: \"ff7a5a9c-2fae-4a88-90bd-f50314fda904\") " pod="openshift-network-operator/iptables-alerter-lrhsn" Jul 14 18:09:57.791105 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.790208 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-var-lib-kubelet\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.791105 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.790270 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d953694d-140b-430a-aa16-6c0d82c8daa6-etc-modprobe-d\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.791105 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.790311 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-systemd-units\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.791105 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.790606 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-env-overrides\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.791105 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.791053 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-ovnkube-config\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.792314 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.792288 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d953694d-140b-430a-aa16-6c0d82c8daa6-etc-tuned\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.792810 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.792791 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e3d8f9e9-f69e-4a3d-b688-ff64a5ec495e-agent-certs\") pod \"konnectivity-agent-sgdnj\" (UID: \"e3d8f9e9-f69e-4a3d-b688-ff64a5ec495e\") " pod="kube-system/konnectivity-agent-sgdnj" Jul 14 18:09:57.793147 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.793130 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-ovn-node-metrics-cert\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.793603 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.793582 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d953694d-140b-430a-aa16-6c0d82c8daa6-tmp\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.798605 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:57.798587 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:09:57.798605 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:57.798607 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:09:57.798808 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:57.798618 2565 projected.go:196] Error preparing data for projected volume kube-api-access-vhhw6 for pod openshift-network-diagnostics/network-check-target-x92qm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:09:57.798808 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:57.798672 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65462b89-c7cc-4843-a6e9-baa2aac800af-kube-api-access-vhhw6 podName:65462b89-c7cc-4843-a6e9-baa2aac800af nodeName:}" failed. No retries permitted until 2026-07-14 18:09:58.29865686 +0000 UTC m=+3.136000120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vhhw6" (UniqueName: "kubernetes.io/projected/65462b89-c7cc-4843-a6e9-baa2aac800af-kube-api-access-vhhw6") pod "network-check-target-x92qm" (UID: "65462b89-c7cc-4843-a6e9-baa2aac800af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:09:57.801617 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.801554 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twzxg\" (UniqueName: \"kubernetes.io/projected/4fa88c56-d6f8-49af-90ec-11f33b911e81-kube-api-access-twzxg\") pod \"aws-ebs-csi-driver-node-8dj76\" (UID: \"4fa88c56-d6f8-49af-90ec-11f33b911e81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dj76" Jul 14 18:09:57.801831 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.801808 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4l4p\" (UniqueName: \"kubernetes.io/projected/a1fc5b88-ec77-4f28-83ff-82725cf2a5e1-kube-api-access-d4l4p\") pod \"network-metrics-daemon-rbgtx\" (UID: \"a1fc5b88-ec77-4f28-83ff-82725cf2a5e1\") " pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:09:57.801911 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.801885 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p48x8\" (UniqueName: \"kubernetes.io/projected/d953694d-140b-430a-aa16-6c0d82c8daa6-kube-api-access-p48x8\") pod \"tuned-nhrbt\" (UID: \"d953694d-140b-430a-aa16-6c0d82c8daa6\") " pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.806882 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.806861 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drkh8\" (UniqueName: \"kubernetes.io/projected/5f0ecb09-5c7f-4709-9b4a-1a6a03580822-kube-api-access-drkh8\") pod \"ovnkube-node-f99bz\" (UID: \"5f0ecb09-5c7f-4709-9b4a-1a6a03580822\") " pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.808449 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.808423 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s2l8\" (UniqueName: \"kubernetes.io/projected/ff7a5a9c-2fae-4a88-90bd-f50314fda904-kube-api-access-9s2l8\") pod \"iptables-alerter-lrhsn\" (UID: \"ff7a5a9c-2fae-4a88-90bd-f50314fda904\") " pod="openshift-network-operator/iptables-alerter-lrhsn" Jul 14 18:09:57.811234 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.809165 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfhcd\" (UniqueName: \"kubernetes.io/projected/87ca2cc2-58d4-4899-8bc1-2b5bd93ebe6f-kube-api-access-mfhcd\") pod \"node-ca-qx9gz\" (UID: \"87ca2cc2-58d4-4899-8bc1-2b5bd93ebe6f\") " pod="openshift-image-registry/node-ca-qx9gz" Jul 14 18:09:57.877624 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.877586 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dmt89" Jul 14 18:09:57.885446 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.885420 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h7jw8" Jul 14 18:09:57.890660 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:57.890630 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e4c55d_de1f_4bda_9f4b_f1dcf46eb2b5.slice/crio-e1fe2ac8f4066ec9fb589e9dfed042357b48d9b6052d64f8fdfa311ad97870bb WatchSource:0}: Error finding container e1fe2ac8f4066ec9fb589e9dfed042357b48d9b6052d64f8fdfa311ad97870bb: Status 404 returned error can't find the container with id e1fe2ac8f4066ec9fb589e9dfed042357b48d9b6052d64f8fdfa311ad97870bb Jul 14 18:09:57.892103 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.892078 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lrhsn" Jul 14 18:09:57.897287 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.897266 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dj76" Jul 14 18:09:57.897732 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:57.897710 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcdcb581_e08c_4d47_90ae_bc03b850964d.slice/crio-e91efdc32cc208f498bbf23ea765202dc6353f743edb1b34407dc1a570ce30cf WatchSource:0}: Error finding container e91efdc32cc208f498bbf23ea765202dc6353f743edb1b34407dc1a570ce30cf: Status 404 returned error can't find the container with id e91efdc32cc208f498bbf23ea765202dc6353f743edb1b34407dc1a570ce30cf Jul 14 18:09:57.903176 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:57.903153 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff7a5a9c_2fae_4a88_90bd_f50314fda904.slice/crio-ec7086089fa397f44e7e2035fa5a16edb478b280677f76cf3bcfbd41e24f91dd WatchSource:0}: Error finding container ec7086089fa397f44e7e2035fa5a16edb478b280677f76cf3bcfbd41e24f91dd: Status 404 returned error can't find the container with id ec7086089fa397f44e7e2035fa5a16edb478b280677f76cf3bcfbd41e24f91dd Jul 14 18:09:57.904676 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.904658 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sgdnj" Jul 14 18:09:57.907885 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:57.907843 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa88c56_d6f8_49af_90ec_11f33b911e81.slice/crio-6db860345c0f78bdb09f077dfa1e0d6c8fe7594ac1d02a05dd54889932fc86d7 WatchSource:0}: Error finding container 6db860345c0f78bdb09f077dfa1e0d6c8fe7594ac1d02a05dd54889932fc86d7: Status 404 returned error can't find the container with id 6db860345c0f78bdb09f077dfa1e0d6c8fe7594ac1d02a05dd54889932fc86d7 Jul 14 18:09:57.910300 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.910284 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qx9gz" Jul 14 18:09:57.916152 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.916133 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" Jul 14 18:09:57.917975 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:57.917949 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d8f9e9_f69e_4a3d_b688_ff64a5ec495e.slice/crio-b7056494535c6ba523cb716769bf8f25a00bf133978cb7b8e0af097941ba551e WatchSource:0}: Error finding container b7056494535c6ba523cb716769bf8f25a00bf133978cb7b8e0af097941ba551e: Status 404 returned error can't find the container with id b7056494535c6ba523cb716769bf8f25a00bf133978cb7b8e0af097941ba551e Jul 14 18:09:57.920319 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:57.920299 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:09:57.922225 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:57.922202 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ca2cc2_58d4_4899_8bc1_2b5bd93ebe6f.slice/crio-30240f18f2ee027b7b262a759fdd74582732661cf62942f738c27aa41a2b9502 WatchSource:0}: Error finding container 30240f18f2ee027b7b262a759fdd74582732661cf62942f738c27aa41a2b9502: Status 404 returned error can't find the container with id 30240f18f2ee027b7b262a759fdd74582732661cf62942f738c27aa41a2b9502 Jul 14 18:09:57.929925 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:57.929886 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd953694d_140b_430a_aa16_6c0d82c8daa6.slice/crio-b68db0f971cd4c00941da362a00121ab0d78387bb2067deaff57189a64f4636a WatchSource:0}: Error finding container b68db0f971cd4c00941da362a00121ab0d78387bb2067deaff57189a64f4636a: Status 404 returned error can't find the container with id b68db0f971cd4c00941da362a00121ab0d78387bb2067deaff57189a64f4636a Jul 14 18:09:57.934323 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:09:57.934132 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f0ecb09_5c7f_4709_9b4a_1a6a03580822.slice/crio-91343d98bebec8d5bcf550c76b38293ca1950c32c62f3c287486df0e2af3b4c8 WatchSource:0}: Error finding container 91343d98bebec8d5bcf550c76b38293ca1950c32c62f3c287486df0e2af3b4c8: Status 404 returned error can't find the container with id 91343d98bebec8d5bcf550c76b38293ca1950c32c62f3c287486df0e2af3b4c8 Jul 14 18:09:58.193048 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:58.192978 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/015d11c8-2009-4c76-815a-c2d87c42e7b5-original-pull-secret\") pod \"global-pull-secret-syncer-6czgz\" (UID: \"015d11c8-2009-4c76-815a-c2d87c42e7b5\") " pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:09:58.193195 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:58.193152 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:09:58.193252 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:58.193227 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/015d11c8-2009-4c76-815a-c2d87c42e7b5-original-pull-secret podName:015d11c8-2009-4c76-815a-c2d87c42e7b5 nodeName:}" failed. No retries permitted until 2026-07-14 18:09:59.193205685 +0000 UTC m=+4.030548926 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/015d11c8-2009-4c76-815a-c2d87c42e7b5-original-pull-secret") pod "global-pull-secret-syncer-6czgz" (UID: "015d11c8-2009-4c76-815a-c2d87c42e7b5") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:09:58.293632 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:58.293592 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1fc5b88-ec77-4f28-83ff-82725cf2a5e1-metrics-certs\") pod \"network-metrics-daemon-rbgtx\" (UID: \"a1fc5b88-ec77-4f28-83ff-82725cf2a5e1\") " pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:09:58.293811 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:58.293775 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:09:58.293872 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:58.293857 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1fc5b88-ec77-4f28-83ff-82725cf2a5e1-metrics-certs podName:a1fc5b88-ec77-4f28-83ff-82725cf2a5e1 nodeName:}" failed. No retries permitted until 2026-07-14 18:09:59.293832041 +0000 UTC m=+4.131175297 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1fc5b88-ec77-4f28-83ff-82725cf2a5e1-metrics-certs") pod "network-metrics-daemon-rbgtx" (UID: "a1fc5b88-ec77-4f28-83ff-82725cf2a5e1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:09:58.394117 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:58.394065 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhhw6\" (UniqueName: \"kubernetes.io/projected/65462b89-c7cc-4843-a6e9-baa2aac800af-kube-api-access-vhhw6\") pod \"network-check-target-x92qm\" (UID: \"65462b89-c7cc-4843-a6e9-baa2aac800af\") " pod="openshift-network-diagnostics/network-check-target-x92qm" Jul 14 18:09:58.394317 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:58.394298 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:09:58.394384 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:58.394327 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:09:58.394384 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:58.394339 2565 projected.go:196] Error preparing data for projected volume kube-api-access-vhhw6 for pod openshift-network-diagnostics/network-check-target-x92qm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:09:58.394506 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:58.394404 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65462b89-c7cc-4843-a6e9-baa2aac800af-kube-api-access-vhhw6 podName:65462b89-c7cc-4843-a6e9-baa2aac800af nodeName:}" failed. No retries permitted until 2026-07-14 18:09:59.394384824 +0000 UTC m=+4.231728064 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vhhw6" (UniqueName: "kubernetes.io/projected/65462b89-c7cc-4843-a6e9-baa2aac800af-kube-api-access-vhhw6") pod "network-check-target-x92qm" (UID: "65462b89-c7cc-4843-a6e9-baa2aac800af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:09:58.640212 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:58.640163 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 18:04:57 +0000 UTC" deadline="2028-03-25 17:48:44.68170378 +0000 UTC" Jul 14 18:09:58.640212 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:58.640196 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14879h38m46.041511553s" Jul 14 18:09:58.693598 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:58.693553 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" event={"ID":"5f0ecb09-5c7f-4709-9b4a-1a6a03580822","Type":"ContainerStarted","Data":"91343d98bebec8d5bcf550c76b38293ca1950c32c62f3c287486df0e2af3b4c8"} Jul 14 18:09:58.694798 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:58.694769 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" event={"ID":"d953694d-140b-430a-aa16-6c0d82c8daa6","Type":"ContainerStarted","Data":"b68db0f971cd4c00941da362a00121ab0d78387bb2067deaff57189a64f4636a"} Jul 14 18:09:58.695973 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:58.695938 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dj76" event={"ID":"4fa88c56-d6f8-49af-90ec-11f33b911e81","Type":"ContainerStarted","Data":"6db860345c0f78bdb09f077dfa1e0d6c8fe7594ac1d02a05dd54889932fc86d7"} Jul 14 18:09:58.697350 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:58.697320 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lrhsn" event={"ID":"ff7a5a9c-2fae-4a88-90bd-f50314fda904","Type":"ContainerStarted","Data":"ec7086089fa397f44e7e2035fa5a16edb478b280677f76cf3bcfbd41e24f91dd"} Jul 14 18:09:58.698666 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:58.698635 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h7jw8" event={"ID":"dcdcb581-e08c-4d47-90ae-bc03b850964d","Type":"ContainerStarted","Data":"e91efdc32cc208f498bbf23ea765202dc6353f743edb1b34407dc1a570ce30cf"} Jul 14 18:09:58.699934 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:58.699907 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qx9gz" event={"ID":"87ca2cc2-58d4-4899-8bc1-2b5bd93ebe6f","Type":"ContainerStarted","Data":"30240f18f2ee027b7b262a759fdd74582732661cf62942f738c27aa41a2b9502"} Jul 14 18:09:58.701187 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:58.701148 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sgdnj" event={"ID":"e3d8f9e9-f69e-4a3d-b688-ff64a5ec495e","Type":"ContainerStarted","Data":"b7056494535c6ba523cb716769bf8f25a00bf133978cb7b8e0af097941ba551e"} Jul 14 18:09:58.702470 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:58.702442 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dmt89" event={"ID":"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5","Type":"ContainerStarted","Data":"e1fe2ac8f4066ec9fb589e9dfed042357b48d9b6052d64f8fdfa311ad97870bb"} Jul 14 18:09:59.200471 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:59.200118 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/015d11c8-2009-4c76-815a-c2d87c42e7b5-original-pull-secret\") pod \"global-pull-secret-syncer-6czgz\" (UID: \"015d11c8-2009-4c76-815a-c2d87c42e7b5\") " pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:09:59.200723 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:59.200540 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:09:59.200723 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:59.200604 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/015d11c8-2009-4c76-815a-c2d87c42e7b5-original-pull-secret podName:015d11c8-2009-4c76-815a-c2d87c42e7b5 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:01.200586287 +0000 UTC m=+6.037929526 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/015d11c8-2009-4c76-815a-c2d87c42e7b5-original-pull-secret") pod "global-pull-secret-syncer-6czgz" (UID: "015d11c8-2009-4c76-815a-c2d87c42e7b5") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:09:59.301204 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:59.301146 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1fc5b88-ec77-4f28-83ff-82725cf2a5e1-metrics-certs\") pod \"network-metrics-daemon-rbgtx\" (UID: \"a1fc5b88-ec77-4f28-83ff-82725cf2a5e1\") " pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:09:59.301351 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:59.301307 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:09:59.301406 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:59.301372 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1fc5b88-ec77-4f28-83ff-82725cf2a5e1-metrics-certs podName:a1fc5b88-ec77-4f28-83ff-82725cf2a5e1 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:01.301352612 +0000 UTC m=+6.138695854 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1fc5b88-ec77-4f28-83ff-82725cf2a5e1-metrics-certs") pod "network-metrics-daemon-rbgtx" (UID: "a1fc5b88-ec77-4f28-83ff-82725cf2a5e1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:09:59.403495 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:59.403454 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhhw6\" (UniqueName: \"kubernetes.io/projected/65462b89-c7cc-4843-a6e9-baa2aac800af-kube-api-access-vhhw6\") pod \"network-check-target-x92qm\" (UID: \"65462b89-c7cc-4843-a6e9-baa2aac800af\") " pod="openshift-network-diagnostics/network-check-target-x92qm" Jul 14 18:09:59.403671 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:59.403657 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:09:59.403732 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:59.403678 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:09:59.403732 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:59.403690 2565 projected.go:196] Error preparing data for projected volume kube-api-access-vhhw6 for pod openshift-network-diagnostics/network-check-target-x92qm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:09:59.403848 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:59.403748 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65462b89-c7cc-4843-a6e9-baa2aac800af-kube-api-access-vhhw6 podName:65462b89-c7cc-4843-a6e9-baa2aac800af nodeName:}" failed. No retries permitted until 2026-07-14 18:10:01.403732095 +0000 UTC m=+6.241075334 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vhhw6" (UniqueName: "kubernetes.io/projected/65462b89-c7cc-4843-a6e9-baa2aac800af-kube-api-access-vhhw6") pod "network-check-target-x92qm" (UID: "65462b89-c7cc-4843-a6e9-baa2aac800af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:09:59.640913 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:59.640856 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 18:04:57 +0000 UTC" deadline="2028-04-06 21:27:21.016142828 +0000 UTC" Jul 14 18:09:59.640913 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:59.640889 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15171h17m21.375258474s" Jul 14 18:09:59.684396 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:59.684371 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x92qm" Jul 14 18:09:59.684498 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:59.684480 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x92qm" podUID="65462b89-c7cc-4843-a6e9-baa2aac800af" Jul 14 18:09:59.684862 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:59.684843 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:09:59.684951 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:59.684932 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6czgz" podUID="015d11c8-2009-4c76-815a-c2d87c42e7b5" Jul 14 18:09:59.685146 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:09:59.685116 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:09:59.685221 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:09:59.685205 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbgtx" podUID="a1fc5b88-ec77-4f28-83ff-82725cf2a5e1" Jul 14 18:10:00.725366 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:00.725285 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-25.ec2.internal" event={"ID":"03b994a435c6ac42ecdd2d02de3d23bb","Type":"ContainerStarted","Data":"0ccecfa26e2b32da358ef5d9cc6b5b1cba2140451b759a8a15d4bcd346445b72"} Jul 14 18:10:00.739621 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:00.739587 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal" event={"ID":"5c5ad4d84a227248b0267edbeaea1f57","Type":"ContainerStarted","Data":"3b35b53d1fb8a694b81ee9db76dd4e2bac90a2b7b5f3301e903a183fab82c94e"} Jul 14 18:10:00.772161 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:00.772075 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-25.ec2.internal" podStartSLOduration=3.772055569 podStartE2EDuration="3.772055569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:00.75012628 +0000 UTC m=+5.587469538" watchObservedRunningTime="2026-07-14 18:10:00.772055569 +0000 UTC m=+5.609398833" Jul 14 18:10:01.218872 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:01.218829 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/015d11c8-2009-4c76-815a-c2d87c42e7b5-original-pull-secret\") pod \"global-pull-secret-syncer-6czgz\" (UID: \"015d11c8-2009-4c76-815a-c2d87c42e7b5\") " pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:10:01.219040 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:01.219003 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:01.219139 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:01.219063 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/015d11c8-2009-4c76-815a-c2d87c42e7b5-original-pull-secret podName:015d11c8-2009-4c76-815a-c2d87c42e7b5 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:05.219044781 +0000 UTC m=+10.056388024 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/015d11c8-2009-4c76-815a-c2d87c42e7b5-original-pull-secret") pod "global-pull-secret-syncer-6czgz" (UID: "015d11c8-2009-4c76-815a-c2d87c42e7b5") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:01.319614 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:01.319575 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1fc5b88-ec77-4f28-83ff-82725cf2a5e1-metrics-certs\") pod \"network-metrics-daemon-rbgtx\" (UID: \"a1fc5b88-ec77-4f28-83ff-82725cf2a5e1\") " pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:10:01.319902 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:01.319791 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:01.319902 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:01.319853 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1fc5b88-ec77-4f28-83ff-82725cf2a5e1-metrics-certs podName:a1fc5b88-ec77-4f28-83ff-82725cf2a5e1 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:05.319834773 +0000 UTC m=+10.157178012 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1fc5b88-ec77-4f28-83ff-82725cf2a5e1-metrics-certs") pod "network-metrics-daemon-rbgtx" (UID: "a1fc5b88-ec77-4f28-83ff-82725cf2a5e1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:01.420462 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:01.420426 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhhw6\" (UniqueName: \"kubernetes.io/projected/65462b89-c7cc-4843-a6e9-baa2aac800af-kube-api-access-vhhw6\") pod \"network-check-target-x92qm\" (UID: \"65462b89-c7cc-4843-a6e9-baa2aac800af\") " pod="openshift-network-diagnostics/network-check-target-x92qm" Jul 14 18:10:01.420768 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:01.420746 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:10:01.420867 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:01.420775 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:10:01.420867 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:01.420802 2565 projected.go:196] Error preparing data for projected volume kube-api-access-vhhw6 for pod openshift-network-diagnostics/network-check-target-x92qm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:01.420963 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:01.420881 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65462b89-c7cc-4843-a6e9-baa2aac800af-kube-api-access-vhhw6 podName:65462b89-c7cc-4843-a6e9-baa2aac800af nodeName:}" failed. No retries permitted until 2026-07-14 18:10:05.420842791 +0000 UTC m=+10.258186043 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vhhw6" (UniqueName: "kubernetes.io/projected/65462b89-c7cc-4843-a6e9-baa2aac800af-kube-api-access-vhhw6") pod "network-check-target-x92qm" (UID: "65462b89-c7cc-4843-a6e9-baa2aac800af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:01.686511 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:01.686477 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:10:01.686853 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:01.686627 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6czgz" podUID="015d11c8-2009-4c76-815a-c2d87c42e7b5" Jul 14 18:10:01.687042 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:01.687017 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:10:01.687169 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:01.687146 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbgtx" podUID="a1fc5b88-ec77-4f28-83ff-82725cf2a5e1" Jul 14 18:10:01.687235 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:01.687207 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x92qm" Jul 14 18:10:01.687300 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:01.687272 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x92qm" podUID="65462b89-c7cc-4843-a6e9-baa2aac800af" Jul 14 18:10:01.744154 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:01.744048 2565 generic.go:356] "Generic (PLEG): container finished" podID="5c5ad4d84a227248b0267edbeaea1f57" containerID="3b35b53d1fb8a694b81ee9db76dd4e2bac90a2b7b5f3301e903a183fab82c94e" exitCode=0 Jul 14 18:10:01.744640 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:01.744156 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal" event={"ID":"5c5ad4d84a227248b0267edbeaea1f57","Type":"ContainerDied","Data":"3b35b53d1fb8a694b81ee9db76dd4e2bac90a2b7b5f3301e903a183fab82c94e"} Jul 14 18:10:03.684804 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:03.684771 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:10:03.685288 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:03.684900 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6czgz" podUID="015d11c8-2009-4c76-815a-c2d87c42e7b5" Jul 14 18:10:03.685288 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:03.685202 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:10:03.685368 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:03.685303 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbgtx" podUID="a1fc5b88-ec77-4f28-83ff-82725cf2a5e1" Jul 14 18:10:03.685368 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:03.685345 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x92qm" Jul 14 18:10:03.685446 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:03.685392 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x92qm" podUID="65462b89-c7cc-4843-a6e9-baa2aac800af" Jul 14 18:10:05.253584 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:05.253543 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/015d11c8-2009-4c76-815a-c2d87c42e7b5-original-pull-secret\") pod \"global-pull-secret-syncer-6czgz\" (UID: \"015d11c8-2009-4c76-815a-c2d87c42e7b5\") " pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:10:05.254033 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:05.253736 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:05.254033 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:05.253812 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/015d11c8-2009-4c76-815a-c2d87c42e7b5-original-pull-secret podName:015d11c8-2009-4c76-815a-c2d87c42e7b5 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:13.253792699 +0000 UTC m=+18.091135936 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/015d11c8-2009-4c76-815a-c2d87c42e7b5-original-pull-secret") pod "global-pull-secret-syncer-6czgz" (UID: "015d11c8-2009-4c76-815a-c2d87c42e7b5") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:05.354786 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:05.354748 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1fc5b88-ec77-4f28-83ff-82725cf2a5e1-metrics-certs\") pod \"network-metrics-daemon-rbgtx\" (UID: \"a1fc5b88-ec77-4f28-83ff-82725cf2a5e1\") " pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:10:05.354972 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:05.354858 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:05.354972 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:05.354933 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1fc5b88-ec77-4f28-83ff-82725cf2a5e1-metrics-certs podName:a1fc5b88-ec77-4f28-83ff-82725cf2a5e1 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:13.354918116 +0000 UTC m=+18.192261374 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1fc5b88-ec77-4f28-83ff-82725cf2a5e1-metrics-certs") pod "network-metrics-daemon-rbgtx" (UID: "a1fc5b88-ec77-4f28-83ff-82725cf2a5e1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:05.456075 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:05.456040 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhhw6\" (UniqueName: \"kubernetes.io/projected/65462b89-c7cc-4843-a6e9-baa2aac800af-kube-api-access-vhhw6\") pod \"network-check-target-x92qm\" (UID: \"65462b89-c7cc-4843-a6e9-baa2aac800af\") " pod="openshift-network-diagnostics/network-check-target-x92qm" Jul 14 18:10:05.456266 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:05.456241 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:10:05.456326 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:05.456269 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:10:05.456326 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:05.456280 2565 projected.go:196] Error preparing data for projected volume kube-api-access-vhhw6 for pod openshift-network-diagnostics/network-check-target-x92qm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:05.456381 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:05.456328 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65462b89-c7cc-4843-a6e9-baa2aac800af-kube-api-access-vhhw6 podName:65462b89-c7cc-4843-a6e9-baa2aac800af nodeName:}" failed. No retries permitted until 2026-07-14 18:10:13.456314588 +0000 UTC m=+18.293657824 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vhhw6" (UniqueName: "kubernetes.io/projected/65462b89-c7cc-4843-a6e9-baa2aac800af-kube-api-access-vhhw6") pod "network-check-target-x92qm" (UID: "65462b89-c7cc-4843-a6e9-baa2aac800af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:05.684864 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:05.683962 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:10:05.684864 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:05.684124 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6czgz" podUID="015d11c8-2009-4c76-815a-c2d87c42e7b5" Jul 14 18:10:05.684864 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:05.684487 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:10:05.684864 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:05.684589 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbgtx" podUID="a1fc5b88-ec77-4f28-83ff-82725cf2a5e1" Jul 14 18:10:05.684864 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:05.683918 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x92qm" Jul 14 18:10:05.684864 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:05.684672 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x92qm" podUID="65462b89-c7cc-4843-a6e9-baa2aac800af" Jul 14 18:10:07.683877 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:07.683841 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:10:07.684362 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:07.683885 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x92qm" Jul 14 18:10:07.684362 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:07.683841 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:10:07.684362 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:07.683977 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6czgz" podUID="015d11c8-2009-4c76-815a-c2d87c42e7b5" Jul 14 18:10:07.684362 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:07.684088 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x92qm" podUID="65462b89-c7cc-4843-a6e9-baa2aac800af" Jul 14 18:10:07.684362 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:07.684175 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbgtx" podUID="a1fc5b88-ec77-4f28-83ff-82725cf2a5e1" Jul 14 18:10:07.756724 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:07.756690 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-ds4bd"] Jul 14 18:10:07.765877 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:07.765850 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ds4bd" Jul 14 18:10:07.769698 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:07.769676 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 14 18:10:07.769827 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:07.769679 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-89hm8\"" Jul 14 18:10:07.770895 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:07.770875 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 14 18:10:07.876755 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:07.876712 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0089b7f3-1d21-4a3d-b5d8-fc1cd289f06e-hosts-file\") pod \"node-resolver-ds4bd\" (UID: \"0089b7f3-1d21-4a3d-b5d8-fc1cd289f06e\") " pod="openshift-dns/node-resolver-ds4bd" Jul 14 18:10:07.876910 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:07.876772 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48fkv\" (UniqueName: \"kubernetes.io/projected/0089b7f3-1d21-4a3d-b5d8-fc1cd289f06e-kube-api-access-48fkv\") pod \"node-resolver-ds4bd\" (UID: \"0089b7f3-1d21-4a3d-b5d8-fc1cd289f06e\") " pod="openshift-dns/node-resolver-ds4bd" Jul 14 18:10:07.876910 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:07.876827 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0089b7f3-1d21-4a3d-b5d8-fc1cd289f06e-tmp-dir\") pod \"node-resolver-ds4bd\" (UID: \"0089b7f3-1d21-4a3d-b5d8-fc1cd289f06e\") " pod="openshift-dns/node-resolver-ds4bd" Jul 14 18:10:07.977570 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:07.977488 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0089b7f3-1d21-4a3d-b5d8-fc1cd289f06e-hosts-file\") pod \"node-resolver-ds4bd\" (UID: \"0089b7f3-1d21-4a3d-b5d8-fc1cd289f06e\") " pod="openshift-dns/node-resolver-ds4bd" Jul 14 18:10:07.977713 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:07.977572 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48fkv\" (UniqueName: \"kubernetes.io/projected/0089b7f3-1d21-4a3d-b5d8-fc1cd289f06e-kube-api-access-48fkv\") pod \"node-resolver-ds4bd\" (UID: \"0089b7f3-1d21-4a3d-b5d8-fc1cd289f06e\") " pod="openshift-dns/node-resolver-ds4bd" Jul 14 18:10:07.977713 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:07.977627 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0089b7f3-1d21-4a3d-b5d8-fc1cd289f06e-tmp-dir\") pod \"node-resolver-ds4bd\" (UID: \"0089b7f3-1d21-4a3d-b5d8-fc1cd289f06e\") " pod="openshift-dns/node-resolver-ds4bd" Jul 14 18:10:07.977713 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:07.977624 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0089b7f3-1d21-4a3d-b5d8-fc1cd289f06e-hosts-file\") pod \"node-resolver-ds4bd\" (UID: \"0089b7f3-1d21-4a3d-b5d8-fc1cd289f06e\") " pod="openshift-dns/node-resolver-ds4bd" Jul 14 18:10:07.977935 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:07.977908 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0089b7f3-1d21-4a3d-b5d8-fc1cd289f06e-tmp-dir\") pod \"node-resolver-ds4bd\" (UID: \"0089b7f3-1d21-4a3d-b5d8-fc1cd289f06e\") " pod="openshift-dns/node-resolver-ds4bd" Jul 14 18:10:07.990284 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:07.990263 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48fkv\" (UniqueName: \"kubernetes.io/projected/0089b7f3-1d21-4a3d-b5d8-fc1cd289f06e-kube-api-access-48fkv\") pod \"node-resolver-ds4bd\" (UID: \"0089b7f3-1d21-4a3d-b5d8-fc1cd289f06e\") " pod="openshift-dns/node-resolver-ds4bd" Jul 14 18:10:08.078756 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:08.078721 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ds4bd" Jul 14 18:10:09.683965 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:09.683934 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:10:09.683965 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:09.683956 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x92qm" Jul 14 18:10:09.684393 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:09.683935 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:10:09.684393 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:09.684063 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbgtx" podUID="a1fc5b88-ec77-4f28-83ff-82725cf2a5e1" Jul 14 18:10:09.684393 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:09.684159 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6czgz" podUID="015d11c8-2009-4c76-815a-c2d87c42e7b5" Jul 14 18:10:09.684393 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:09.684245 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x92qm" podUID="65462b89-c7cc-4843-a6e9-baa2aac800af" Jul 14 18:10:11.683995 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:11.683962 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:10:11.684475 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:11.684086 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6czgz" podUID="015d11c8-2009-4c76-815a-c2d87c42e7b5" Jul 14 18:10:11.684549 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:11.684528 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:10:11.684665 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:11.684645 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbgtx" podUID="a1fc5b88-ec77-4f28-83ff-82725cf2a5e1" Jul 14 18:10:11.684840 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:11.684809 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x92qm" Jul 14 18:10:11.684971 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:11.684913 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x92qm" podUID="65462b89-c7cc-4843-a6e9-baa2aac800af" Jul 14 18:10:13.318368 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:13.318322 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/015d11c8-2009-4c76-815a-c2d87c42e7b5-original-pull-secret\") pod \"global-pull-secret-syncer-6czgz\" (UID: \"015d11c8-2009-4c76-815a-c2d87c42e7b5\") " pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:10:13.318834 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:13.318469 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:13.318834 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:13.318543 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/015d11c8-2009-4c76-815a-c2d87c42e7b5-original-pull-secret podName:015d11c8-2009-4c76-815a-c2d87c42e7b5 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:29.318523322 +0000 UTC m=+34.155866577 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/015d11c8-2009-4c76-815a-c2d87c42e7b5-original-pull-secret") pod "global-pull-secret-syncer-6czgz" (UID: "015d11c8-2009-4c76-815a-c2d87c42e7b5") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:13.419070 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:13.419033 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1fc5b88-ec77-4f28-83ff-82725cf2a5e1-metrics-certs\") pod \"network-metrics-daemon-rbgtx\" (UID: \"a1fc5b88-ec77-4f28-83ff-82725cf2a5e1\") " pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:10:13.419268 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:13.419177 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:13.419268 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:13.419248 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1fc5b88-ec77-4f28-83ff-82725cf2a5e1-metrics-certs podName:a1fc5b88-ec77-4f28-83ff-82725cf2a5e1 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:29.419231085 +0000 UTC m=+34.256574355 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1fc5b88-ec77-4f28-83ff-82725cf2a5e1-metrics-certs") pod "network-metrics-daemon-rbgtx" (UID: "a1fc5b88-ec77-4f28-83ff-82725cf2a5e1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:13.519871 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:13.519832 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhhw6\" (UniqueName: \"kubernetes.io/projected/65462b89-c7cc-4843-a6e9-baa2aac800af-kube-api-access-vhhw6\") pod \"network-check-target-x92qm\" (UID: \"65462b89-c7cc-4843-a6e9-baa2aac800af\") " pod="openshift-network-diagnostics/network-check-target-x92qm" Jul 14 18:10:13.520059 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:13.520017 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:10:13.520059 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:13.520041 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:10:13.520059 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:13.520054 2565 projected.go:196] Error preparing data for projected volume kube-api-access-vhhw6 for pod openshift-network-diagnostics/network-check-target-x92qm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:13.520239 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:13.520131 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65462b89-c7cc-4843-a6e9-baa2aac800af-kube-api-access-vhhw6 podName:65462b89-c7cc-4843-a6e9-baa2aac800af nodeName:}" failed. No retries permitted until 2026-07-14 18:10:29.52011235 +0000 UTC m=+34.357455618 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vhhw6" (UniqueName: "kubernetes.io/projected/65462b89-c7cc-4843-a6e9-baa2aac800af-kube-api-access-vhhw6") pod "network-check-target-x92qm" (UID: "65462b89-c7cc-4843-a6e9-baa2aac800af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:13.683827 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:13.683738 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:10:13.683827 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:13.683776 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x92qm" Jul 14 18:10:13.683827 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:13.683756 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:10:13.684075 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:13.683861 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbgtx" podUID="a1fc5b88-ec77-4f28-83ff-82725cf2a5e1" Jul 14 18:10:13.684075 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:13.683916 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6czgz" podUID="015d11c8-2009-4c76-815a-c2d87c42e7b5" Jul 14 18:10:13.684075 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:13.684016 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x92qm" podUID="65462b89-c7cc-4843-a6e9-baa2aac800af" Jul 14 18:10:15.687116 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:15.687078 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:10:15.687518 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:15.687219 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:10:15.687518 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:15.687231 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbgtx" podUID="a1fc5b88-ec77-4f28-83ff-82725cf2a5e1" Jul 14 18:10:15.687518 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:15.687248 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x92qm" Jul 14 18:10:15.687518 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:15.687307 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6czgz" podUID="015d11c8-2009-4c76-815a-c2d87c42e7b5" Jul 14 18:10:15.687518 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:15.687380 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x92qm" podUID="65462b89-c7cc-4843-a6e9-baa2aac800af" Jul 14 18:10:17.687237 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:17.686788 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x92qm" Jul 14 18:10:17.687237 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:17.686818 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:10:17.687891 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:17.687246 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x92qm" podUID="65462b89-c7cc-4843-a6e9-baa2aac800af" Jul 14 18:10:17.687891 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:17.686846 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:10:17.687891 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:17.687311 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6czgz" podUID="015d11c8-2009-4c76-815a-c2d87c42e7b5" Jul 14 18:10:17.687891 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:17.687387 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbgtx" podUID="a1fc5b88-ec77-4f28-83ff-82725cf2a5e1" Jul 14 18:10:17.775830 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:17.775797 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal" event={"ID":"5c5ad4d84a227248b0267edbeaea1f57","Type":"ContainerStarted","Data":"4a3831eabf8fb5801146458c5d050274206468b5aa9d294d22be39aee3341310"} Jul 14 18:10:17.778632 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:17.778595 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f99bz_5f0ecb09-5c7f-4709-9b4a-1a6a03580822/ovn-acl-logging/0.log" Jul 14 18:10:17.778923 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:17.778907 2565 generic.go:356] "Generic (PLEG): container finished" podID="5f0ecb09-5c7f-4709-9b4a-1a6a03580822" containerID="10558ecc0a64c67fafaccd0d1092819ec21722134c469abd14e16e4f12c74c9e" exitCode=1 Jul 14 18:10:17.779010 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:17.778945 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" event={"ID":"5f0ecb09-5c7f-4709-9b4a-1a6a03580822","Type":"ContainerStarted","Data":"28215d52ae364a7942bd0092acf8cd83de02130eac924fb5dd80e729b37c38e5"} Jul 14 18:10:17.779053 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:17.779011 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" event={"ID":"5f0ecb09-5c7f-4709-9b4a-1a6a03580822","Type":"ContainerStarted","Data":"67d20995981f6660a14c80f9538cf877c800178a2c6f9259b80bfb3f1825a775"} Jul 14 18:10:17.779083 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:17.779061 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" event={"ID":"5f0ecb09-5c7f-4709-9b4a-1a6a03580822","Type":"ContainerStarted","Data":"f6627909d5de7a12d40d90d88e3cb6aea3a6fb75db49409fcc8d22912c9bd822"} Jul 14 18:10:17.779137 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:17.779128 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" event={"ID":"5f0ecb09-5c7f-4709-9b4a-1a6a03580822","Type":"ContainerStarted","Data":"79a9ea1826ae6996d7830efd401b0d8109363a68dd471d204dbb2694b25ffaea"} Jul 14 18:10:17.779191 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:17.779178 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" event={"ID":"5f0ecb09-5c7f-4709-9b4a-1a6a03580822","Type":"ContainerDied","Data":"10558ecc0a64c67fafaccd0d1092819ec21722134c469abd14e16e4f12c74c9e"} Jul 14 18:10:17.779238 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:17.779228 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" event={"ID":"5f0ecb09-5c7f-4709-9b4a-1a6a03580822","Type":"ContainerStarted","Data":"73bdccac266135e093eeb58c6a1fe40d55936cb94754b3fe88e8a4e8091cc9f5"} Jul 14 18:10:17.780383 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:17.780362 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" event={"ID":"d953694d-140b-430a-aa16-6c0d82c8daa6","Type":"ContainerStarted","Data":"829193b6d55f9b91b2a11b4c70f461da09ee690c0b4527312432af7dbec27e2e"} Jul 14 18:10:17.781592 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:17.781574 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dj76" event={"ID":"4fa88c56-d6f8-49af-90ec-11f33b911e81","Type":"ContainerStarted","Data":"e68024640321b1caa059ffea487d55262b06f4269a26d509e638d5198af8bcfd"} Jul 14 18:10:17.782769 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:17.782750 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h7jw8" event={"ID":"dcdcb581-e08c-4d47-90ae-bc03b850964d","Type":"ContainerStarted","Data":"fe0441a20e72d74e1bc5ed347d7d47f6c9bacdf231af72e27047ba2d9c2d8b41"} Jul 14 18:10:17.784032 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:17.784015 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ds4bd" event={"ID":"0089b7f3-1d21-4a3d-b5d8-fc1cd289f06e","Type":"ContainerStarted","Data":"4ffc776216c93a7b7b515d8e4e40ab6abc509c3b1a636814a7457a35a2affcc1"} Jul 14 18:10:17.784142 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:17.784044 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ds4bd" event={"ID":"0089b7f3-1d21-4a3d-b5d8-fc1cd289f06e","Type":"ContainerStarted","Data":"87bda0b0b6ce7db802c3b6c39817c1a01a05cdd5a4f752ba58dc4e278ac8cc30"} Jul 14 18:10:17.785452 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:17.785428 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qx9gz" event={"ID":"87ca2cc2-58d4-4899-8bc1-2b5bd93ebe6f","Type":"ContainerStarted","Data":"ce2b0f87f096d6f1953e8914cb42310ace49b88b225b1b42becd934005e24e40"} Jul 14 18:10:17.786798 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:17.786777 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sgdnj" event={"ID":"e3d8f9e9-f69e-4a3d-b688-ff64a5ec495e","Type":"ContainerStarted","Data":"57a3d92e0dce1c0f85418d6ee79dca863d77ae4239e533d626401d24796eca10"} Jul 14 18:10:17.788207 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:17.788188 2565 generic.go:356] "Generic (PLEG): container finished" podID="42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5" containerID="694c6b65b9bf28c19344c9389f9fb200a447b5f464a0313d3b1e0c855d46318d" exitCode=0 Jul 14 18:10:17.788283 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:17.788217 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dmt89" event={"ID":"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5","Type":"ContainerDied","Data":"694c6b65b9bf28c19344c9389f9fb200a447b5f464a0313d3b1e0c855d46318d"} Jul 14 18:10:17.862498 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:17.862452 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-25.ec2.internal" podStartSLOduration=20.862439001 podStartE2EDuration="20.862439001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:17.826432786 +0000 UTC m=+22.663776042" watchObservedRunningTime="2026-07-14 18:10:17.862439001 +0000 UTC m=+22.699782255" Jul 14 18:10:17.862702 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:17.862673 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sgdnj" podStartSLOduration=2.994420004 podStartE2EDuration="21.862665022s" totalImagesPullingTime="18.868245018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:09:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:17.862663915 +0000 UTC m=+22.700007172" watchObservedRunningTime="2026-07-14 18:10:17.862665022 +0000 UTC m=+22.700008281" Jul 14 18:10:17.925093 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:17.925051 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ds4bd" podStartSLOduration=10.925037919 podStartE2EDuration="10.925037919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:17.894303431 +0000 UTC m=+22.731646686" watchObservedRunningTime="2026-07-14 18:10:17.925037919 +0000 UTC m=+22.762381199" Jul 14 18:10:17.979608 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:17.979524 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nhrbt" podStartSLOduration=3.101173756 podStartE2EDuration="21.9795104s" totalImagesPullingTime="18.878336644s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:09:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:17.979244588 +0000 UTC m=+22.816587845" watchObservedRunningTime="2026-07-14 18:10:17.9795104 +0000 UTC m=+22.816853658" Jul 14 18:10:17.979732 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:17.979603 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qx9gz" podStartSLOduration=3.101225252 podStartE2EDuration="21.97959952s" totalImagesPullingTime="18.878374268s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:09:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:17.949088844 +0000 UTC m=+22.786432101" watchObservedRunningTime="2026-07-14 18:10:17.97959952 +0000 UTC m=+22.816942778" Jul 14 18:10:18.011753 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:18.011701 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-h7jw8" podStartSLOduration=4.027358772 podStartE2EDuration="23.011688144s" totalImagesPullingTime="18.984329372s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:09:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:18.01135868 +0000 UTC m=+22.848701938" watchObservedRunningTime="2026-07-14 18:10:18.011688144 +0000 UTC m=+22.849031401" Jul 14 18:10:18.792149 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:18.792112 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lrhsn" event={"ID":"ff7a5a9c-2fae-4a88-90bd-f50314fda904","Type":"ContainerStarted","Data":"5245a6b9390384ef43fce905f0265f473ad257bd0d0d82e9f7f4f6ad81ece9f6"} Jul 14 18:10:19.465345 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:19.464995 2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 14 18:10:19.639975 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:19.639871 2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T18:10:19.465315186Z","UUID":"54b161c8-9c3b-4f46-96a1-b13515ee8d8c","Handler":null,"Name":"","Endpoint":""} Jul 14 18:10:19.641639 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:19.641617 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 14 18:10:19.641746 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:19.641658 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 14 18:10:19.684764 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:19.684737 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:10:19.684937 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:19.684743 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x92qm" Jul 14 18:10:19.684937 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:19.684853 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6czgz" podUID="015d11c8-2009-4c76-815a-c2d87c42e7b5" Jul 14 18:10:19.685043 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:19.684743 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:10:19.685043 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:19.684928 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x92qm" podUID="65462b89-c7cc-4843-a6e9-baa2aac800af" Jul 14 18:10:19.685043 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:19.685026 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbgtx" podUID="a1fc5b88-ec77-4f28-83ff-82725cf2a5e1" Jul 14 18:10:19.797153 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:19.797089 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dj76" event={"ID":"4fa88c56-d6f8-49af-90ec-11f33b911e81","Type":"ContainerStarted","Data":"667dadc1574a414ce7b6e2df2f08820eea77186754913c0330a7ce335d9331ba"} Jul 14 18:10:19.800709 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:19.800687 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f99bz_5f0ecb09-5c7f-4709-9b4a-1a6a03580822/ovn-acl-logging/0.log" Jul 14 18:10:19.801140 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:19.801115 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" event={"ID":"5f0ecb09-5c7f-4709-9b4a-1a6a03580822","Type":"ContainerStarted","Data":"7cf5f2754f62dac5f1d3487efccdca31ee2a57d3aa3f1203efe2c5a673966e46"} Jul 14 18:10:20.806052 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:20.806013 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dj76" event={"ID":"4fa88c56-d6f8-49af-90ec-11f33b911e81","Type":"ContainerStarted","Data":"5d8996d27ed35a86dadc9e7bad04c5eac51e3344761435e411d0a49ee7aab75b"} Jul 14 18:10:20.828434 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:20.828377 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lrhsn" podStartSLOduration=5.931130157 podStartE2EDuration="24.828355251s" totalImagesPullingTime="18.897225094s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:09:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:18.825841463 +0000 UTC m=+23.663184720" watchObservedRunningTime="2026-07-14 18:10:20.828355251 +0000 UTC m=+25.665698498" Jul 14 18:10:20.829191 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:20.829158 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dj76" podStartSLOduration=2.683662825 podStartE2EDuration="24.829142611s" totalImagesPullingTime="22.145479786s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:09:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:20.828314774 +0000 UTC m=+25.665658032" watchObservedRunningTime="2026-07-14 18:10:20.829142611 +0000 UTC m=+25.666485869" Jul 14 18:10:21.684192 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:21.684158 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x92qm" Jul 14 18:10:21.684365 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:21.684158 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:10:21.684365 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:21.684273 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x92qm" podUID="65462b89-c7cc-4843-a6e9-baa2aac800af" Jul 14 18:10:21.684365 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:21.684158 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:10:21.684455 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:21.684402 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbgtx" podUID="a1fc5b88-ec77-4f28-83ff-82725cf2a5e1" Jul 14 18:10:21.684455 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:21.684436 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6czgz" podUID="015d11c8-2009-4c76-815a-c2d87c42e7b5" Jul 14 18:10:22.196598 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:22.196338 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sgdnj" Jul 14 18:10:22.197089 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:22.196892 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sgdnj" Jul 14 18:10:22.812774 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:22.812742 2565 generic.go:356] "Generic (PLEG): container finished" podID="42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5" containerID="5a8c517a3323034b49e7246e973407167354461dc8cf9a5f8e6e51cc9a22d12d" exitCode=0 Jul 14 18:10:22.812956 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:22.812837 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dmt89" event={"ID":"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5","Type":"ContainerDied","Data":"5a8c517a3323034b49e7246e973407167354461dc8cf9a5f8e6e51cc9a22d12d"} Jul 14 18:10:22.816149 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:22.816132 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f99bz_5f0ecb09-5c7f-4709-9b4a-1a6a03580822/ovn-acl-logging/0.log" Jul 14 18:10:22.816594 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:22.816562 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" event={"ID":"5f0ecb09-5c7f-4709-9b4a-1a6a03580822","Type":"ContainerStarted","Data":"a1844c89e5996b8f066b3db1edadb483309f843dc1600a3154894e071e9b695e"} Jul 14 18:10:22.816788 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:22.816771 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sgdnj" Jul 14 18:10:22.817005 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:22.816991 2565 scope.go:117] "RemoveContainer" containerID="10558ecc0a64c67fafaccd0d1092819ec21722134c469abd14e16e4f12c74c9e" Jul 14 18:10:22.817191 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:22.817177 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sgdnj" Jul 14 18:10:23.683972 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:23.683949 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:10:23.684359 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:23.683950 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:10:23.684359 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:23.684045 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbgtx" podUID="a1fc5b88-ec77-4f28-83ff-82725cf2a5e1" Jul 14 18:10:23.684359 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:23.683950 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x92qm" Jul 14 18:10:23.684359 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:23.684180 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6czgz" podUID="015d11c8-2009-4c76-815a-c2d87c42e7b5" Jul 14 18:10:23.684359 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:23.684268 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x92qm" podUID="65462b89-c7cc-4843-a6e9-baa2aac800af" Jul 14 18:10:23.820812 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:23.820782 2565 generic.go:356] "Generic (PLEG): container finished" podID="42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5" containerID="511924a32b987a08b17b6a9e38fc4e305081378e708279ff3ce4c1178036ee7b" exitCode=0 Jul 14 18:10:23.820994 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:23.820862 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dmt89" event={"ID":"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5","Type":"ContainerDied","Data":"511924a32b987a08b17b6a9e38fc4e305081378e708279ff3ce4c1178036ee7b"} Jul 14 18:10:23.825171 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:23.825069 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f99bz_5f0ecb09-5c7f-4709-9b4a-1a6a03580822/ovn-acl-logging/0.log" Jul 14 18:10:23.825528 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:23.825507 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" event={"ID":"5f0ecb09-5c7f-4709-9b4a-1a6a03580822","Type":"ContainerStarted","Data":"02468ba28824aab8e282610418908d460e65ffa7320b516bff1a9fc1de31fb61"} Jul 14 18:10:23.825636 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:23.825593 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:10:23.825691 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:23.825657 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:10:23.825851 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:23.825825 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:10:23.841505 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:23.841467 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:10:23.841797 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:23.841602 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:10:23.890730 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:23.890637 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" podStartSLOduration=8.86469788 podStartE2EDuration="27.890599296s" totalImagesPullingTime="19.025901416s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:09:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:23.890409115 +0000 UTC m=+28.727752372" watchObservedRunningTime="2026-07-14 18:10:23.890599296 +0000 UTC m=+28.727942554" Jul 14 18:10:24.138217 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:24.138188 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x92qm"] Jul 14 18:10:24.138357 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:24.138277 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x92qm" Jul 14 18:10:24.138400 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:24.138352 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x92qm" podUID="65462b89-c7cc-4843-a6e9-baa2aac800af" Jul 14 18:10:24.140351 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:24.140323 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6czgz"] Jul 14 18:10:24.140465 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:24.140410 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:10:24.140516 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:24.140494 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6czgz" podUID="015d11c8-2009-4c76-815a-c2d87c42e7b5" Jul 14 18:10:24.144318 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:24.144261 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rbgtx"] Jul 14 18:10:24.144427 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:24.144331 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:10:24.144427 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:24.144404 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbgtx" podUID="a1fc5b88-ec77-4f28-83ff-82725cf2a5e1" Jul 14 18:10:24.830013 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:24.829831 2565 generic.go:356] "Generic (PLEG): container finished" podID="42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5" containerID="392fd44ace40f70e1db9eeafc64231f5a5feac767215d74c435b4864821bdbbc" exitCode=0 Jul 14 18:10:24.830490 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:24.830113 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dmt89" event={"ID":"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5","Type":"ContainerDied","Data":"392fd44ace40f70e1db9eeafc64231f5a5feac767215d74c435b4864821bdbbc"} Jul 14 18:10:25.684984 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:25.684947 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:10:25.685171 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:25.685053 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6czgz" podUID="015d11c8-2009-4c76-815a-c2d87c42e7b5" Jul 14 18:10:25.685171 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:25.685137 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:10:25.685297 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:25.685201 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbgtx" podUID="a1fc5b88-ec77-4f28-83ff-82725cf2a5e1" Jul 14 18:10:25.685297 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:25.685264 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x92qm" Jul 14 18:10:25.685392 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:25.685370 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x92qm" podUID="65462b89-c7cc-4843-a6e9-baa2aac800af" Jul 14 18:10:27.684305 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:27.684274 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x92qm" Jul 14 18:10:27.684305 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:27.684292 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:10:27.685123 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:27.684394 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x92qm" podUID="65462b89-c7cc-4843-a6e9-baa2aac800af" Jul 14 18:10:27.685123 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:27.684877 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:10:27.685123 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:27.685016 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6czgz" podUID="015d11c8-2009-4c76-815a-c2d87c42e7b5" Jul 14 18:10:27.685235 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:27.685127 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbgtx" podUID="a1fc5b88-ec77-4f28-83ff-82725cf2a5e1" Jul 14 18:10:29.335341 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.335314 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/015d11c8-2009-4c76-815a-c2d87c42e7b5-original-pull-secret\") pod \"global-pull-secret-syncer-6czgz\" (UID: \"015d11c8-2009-4c76-815a-c2d87c42e7b5\") " pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:10:29.335807 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:29.335450 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:29.335807 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:29.335509 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/015d11c8-2009-4c76-815a-c2d87c42e7b5-original-pull-secret podName:015d11c8-2009-4c76-815a-c2d87c42e7b5 nodeName:}" failed. No retries permitted until 2026-07-14 18:11:01.335494836 +0000 UTC m=+66.172838076 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/015d11c8-2009-4c76-815a-c2d87c42e7b5-original-pull-secret") pod "global-pull-secret-syncer-6czgz" (UID: "015d11c8-2009-4c76-815a-c2d87c42e7b5") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:29.435850 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.435826 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1fc5b88-ec77-4f28-83ff-82725cf2a5e1-metrics-certs\") pod \"network-metrics-daemon-rbgtx\" (UID: \"a1fc5b88-ec77-4f28-83ff-82725cf2a5e1\") " pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:10:29.435977 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:29.435963 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:29.436034 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:29.436025 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1fc5b88-ec77-4f28-83ff-82725cf2a5e1-metrics-certs podName:a1fc5b88-ec77-4f28-83ff-82725cf2a5e1 nodeName:}" failed. No retries permitted until 2026-07-14 18:11:01.436008751 +0000 UTC m=+66.273351999 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1fc5b88-ec77-4f28-83ff-82725cf2a5e1-metrics-certs") pod "network-metrics-daemon-rbgtx" (UID: "a1fc5b88-ec77-4f28-83ff-82725cf2a5e1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:29.472800 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.472779 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-25.ec2.internal" event="NodeReady" Jul 14 18:10:29.472902 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.472895 2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 14 18:10:29.531959 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.531937 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f46bf986b-hh2fk"] Jul 14 18:10:29.536908 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.536888 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhhw6\" (UniqueName: \"kubernetes.io/projected/65462b89-c7cc-4843-a6e9-baa2aac800af-kube-api-access-vhhw6\") pod \"network-check-target-x92qm\" (UID: \"65462b89-c7cc-4843-a6e9-baa2aac800af\") " pod="openshift-network-diagnostics/network-check-target-x92qm" Jul 14 18:10:29.537018 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:29.537006 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:10:29.537058 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:29.537024 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:10:29.537058 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:29.537032 2565 projected.go:196] Error preparing data for projected volume kube-api-access-vhhw6 for pod openshift-network-diagnostics/network-check-target-x92qm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:29.537141 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:29.537071 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65462b89-c7cc-4843-a6e9-baa2aac800af-kube-api-access-vhhw6 podName:65462b89-c7cc-4843-a6e9-baa2aac800af nodeName:}" failed. No retries permitted until 2026-07-14 18:11:01.53706054 +0000 UTC m=+66.374403775 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vhhw6" (UniqueName: "kubernetes.io/projected/65462b89-c7cc-4843-a6e9-baa2aac800af-kube-api-access-vhhw6") pod "network-check-target-x92qm" (UID: "65462b89-c7cc-4843-a6e9-baa2aac800af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:29.546347 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.546329 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:10:29.546347 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.546339 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-mqqk9"] Jul 14 18:10:29.549865 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.549849 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 14 18:10:29.550082 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.550063 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 14 18:10:29.551697 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.551678 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 14 18:10:29.552083 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.552066 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kfgwx\"" Jul 14 18:10:29.572425 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.572401 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-mqqk9"] Jul 14 18:10:29.572571 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.572444 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f46bf986b-hh2fk"] Jul 14 18:10:29.572571 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.572473 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7jp42"] Jul 14 18:10:29.572674 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.572572 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-mqqk9" Jul 14 18:10:29.572871 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.572780 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 14 18:10:29.575503 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.575305 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vkvm4\"" Jul 14 18:10:29.575503 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.575367 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 14 18:10:29.576147 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.576129 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 14 18:10:29.576235 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.576193 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 14 18:10:29.576497 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.576477 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 14 18:10:29.584074 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.584056 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 14 18:10:29.594698 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.594626 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-qldqp"] Jul 14 18:10:29.595088 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.595065 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7jp42" Jul 14 18:10:29.600206 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.600179 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 14 18:10:29.600206 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.600197 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 14 18:10:29.600347 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.600215 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-n6z24\"" Jul 14 18:10:29.600347 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.600188 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 14 18:10:29.602024 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.602009 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 14 18:10:29.616983 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.616706 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-n4l4d"] Jul 14 18:10:29.617078 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.617021 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-qldqp" Jul 14 18:10:29.621367 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.621345 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 14 18:10:29.624423 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.621432 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-nmd7g\"" Jul 14 18:10:29.624546 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.621532 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 14 18:10:29.636191 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.636169 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5977bb4ff6-wbzmj"] Jul 14 18:10:29.636280 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.636268 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-n4l4d" Jul 14 18:10:29.637464 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.637444 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-image-registry-private-configuration\") pod \"image-registry-5f46bf986b-hh2fk\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:10:29.637647 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.637472 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-registry-tls\") pod \"image-registry-5f46bf986b-hh2fk\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:10:29.637647 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.637489 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-ca-trust-extracted\") pod \"image-registry-5f46bf986b-hh2fk\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:10:29.637647 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.637514 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dljfz\" (UniqueName: \"kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-kube-api-access-dljfz\") pod \"image-registry-5f46bf986b-hh2fk\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:10:29.637647 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.637549 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-registry-certificates\") pod \"image-registry-5f46bf986b-hh2fk\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:10:29.637647 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.637633 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-trusted-ca\") pod \"image-registry-5f46bf986b-hh2fk\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:10:29.637920 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.637692 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-installation-pull-secrets\") pod \"image-registry-5f46bf986b-hh2fk\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:10:29.637920 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.637755 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-bound-sa-token\") pod \"image-registry-5f46bf986b-hh2fk\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:10:29.639307 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.639286 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 14 18:10:29.639581 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.639385 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 14 18:10:29.639581 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.639420 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-lk8cm\"" Jul 14 18:10:29.639581 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.639497 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 14 18:10:29.662400 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.662347 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6k9qk"] Jul 14 18:10:29.662495 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.662466 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:10:29.665380 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.665362 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 14 18:10:29.665678 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.665664 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 14 18:10:29.665903 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.665886 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 14 18:10:29.665903 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.665899 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-np2nj\"" Jul 14 18:10:29.666014 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.665903 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 14 18:10:29.666014 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.665892 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 14 18:10:29.666214 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.666202 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 14 18:10:29.676476 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.676460 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-wqpt2"] Jul 14 18:10:29.676580 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.676566 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6k9qk" Jul 14 18:10:29.679150 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.679132 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 14 18:10:29.679425 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.679412 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 14 18:10:29.679474 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.679450 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zklb7\"" Jul 14 18:10:29.692941 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.692920 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-wqpt2" Jul 14 18:10:29.693026 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.692970 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-qldqp"] Jul 14 18:10:29.693087 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.693072 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:10:29.693496 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.693245 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x92qm" Jul 14 18:10:29.693496 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.693373 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:10:29.696348 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.696311 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 14 18:10:29.696431 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.696376 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 18:10:29.696431 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.696419 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 18:10:29.696542 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.696445 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-dltbq\"" Jul 14 18:10:29.696542 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.696325 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5vgxh\"" Jul 14 18:10:29.696641 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.696573 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 14 18:10:29.696641 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.696315 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 18:10:29.696786 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.696710 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 14 18:10:29.696786 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.696729 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 18:10:29.697089 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.696797 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jkhfz\"" Jul 14 18:10:29.697089 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.696900 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 14 18:10:29.697089 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.696978 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7jp42"] Jul 14 18:10:29.697089 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.697008 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-wqpt2"] Jul 14 18:10:29.697089 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.697021 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5977bb4ff6-wbzmj"] Jul 14 18:10:29.697089 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.697035 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-7kfjz"] Jul 14 18:10:29.701917 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.701900 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 14 18:10:29.711918 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.711903 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-7kfjz" Jul 14 18:10:29.711918 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.711911 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-n4l4d"] Jul 14 18:10:29.712049 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.711962 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-lgbnj"] Jul 14 18:10:29.719373 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.718942 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 14 18:10:29.719466 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.719375 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 14 18:10:29.719466 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.719388 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 14 18:10:29.719466 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.719441 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-j9dgq\"" Jul 14 18:10:29.719466 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.719452 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 14 18:10:29.729890 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.729873 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-vwp4s"] Jul 14 18:10:29.729983 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.729970 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-lgbnj" Jul 14 18:10:29.732946 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.732928 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 14 18:10:29.733180 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.733167 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-92j4h\"" Jul 14 18:10:29.733456 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.733444 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 14 18:10:29.738819 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.738801 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zqr7\" (UniqueName: \"kubernetes.io/projected/1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452-kube-api-access-6zqr7\") pod \"cluster-monitoring-operator-5f7f494dd4-7jp42\" (UID: \"1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7jp42" Jul 14 18:10:29.738906 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.738841 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-image-registry-private-configuration\") pod \"image-registry-5f46bf986b-hh2fk\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:10:29.738906 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.738860 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-7jp42\" (UID: \"1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7jp42" Jul 14 18:10:29.739016 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.738902 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29cfec31-7adb-41a3-965d-11c1d199b9c3-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-mqqk9\" (UID: \"29cfec31-7adb-41a3-965d-11c1d199b9c3\") " pod="openshift-insights/insights-operator-68d9dd94cf-mqqk9" Jul 14 18:10:29.739016 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.738927 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-default-certificate\") pod \"router-default-5977bb4ff6-wbzmj\" (UID: \"7b2746f9-dc43-4311-aa5b-ad3bc351deb3\") " pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:10:29.739016 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.738948 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-registry-tls\") pod \"image-registry-5f46bf986b-hh2fk\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:10:29.739016 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.738982 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-ca-trust-extracted\") pod \"image-registry-5f46bf986b-hh2fk\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:10:29.739016 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.739016 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6vfc\" (UniqueName: \"kubernetes.io/projected/938c2572-16f6-4f69-90e7-f33f7e46d4b4-kube-api-access-q6vfc\") pod \"volume-data-source-validator-78495597f6-qldqp\" (UID: \"938c2572-16f6-4f69-90e7-f33f7e46d4b4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-qldqp" Jul 14 18:10:29.739257 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:29.739019 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 18:10:29.739257 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:29.739033 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f46bf986b-hh2fk: secret "image-registry-tls" not found Jul 14 18:10:29.739257 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.739037 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29cfec31-7adb-41a3-965d-11c1d199b9c3-tmp\") pod \"insights-operator-68d9dd94cf-mqqk9\" (UID: \"29cfec31-7adb-41a3-965d-11c1d199b9c3\") " pod="openshift-insights/insights-operator-68d9dd94cf-mqqk9" Jul 14 18:10:29.739257 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.739169 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dljfz\" (UniqueName: \"kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-kube-api-access-dljfz\") pod \"image-registry-5f46bf986b-hh2fk\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:10:29.739257 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:29.739185 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-registry-tls podName:bd03d2f0-2403-414f-b02e-ec3c5e7fb214 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:30.239166994 +0000 UTC m=+35.076510247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-registry-tls") pod "image-registry-5f46bf986b-hh2fk" (UID: "bd03d2f0-2403-414f-b02e-ec3c5e7fb214") : secret "image-registry-tls" not found Jul 14 18:10:29.739257 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.739210 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49p8l\" (UniqueName: \"kubernetes.io/projected/5d052fcc-79b5-45cd-a58e-6bb582e25ab8-kube-api-access-49p8l\") pod \"cluster-samples-operator-65db858894-n4l4d\" (UID: \"5d052fcc-79b5-45cd-a58e-6bb582e25ab8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-n4l4d" Jul 14 18:10:29.739257 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.739239 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-registry-certificates\") pod \"image-registry-5f46bf986b-hh2fk\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:10:29.739552 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.739261 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/29cfec31-7adb-41a3-965d-11c1d199b9c3-snapshots\") pod \"insights-operator-68d9dd94cf-mqqk9\" (UID: \"29cfec31-7adb-41a3-965d-11c1d199b9c3\") " pod="openshift-insights/insights-operator-68d9dd94cf-mqqk9" Jul 14 18:10:29.739552 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.739284 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhk7z\" (UniqueName: \"kubernetes.io/projected/29cfec31-7adb-41a3-965d-11c1d199b9c3-kube-api-access-bhk7z\") pod \"insights-operator-68d9dd94cf-mqqk9\" (UID: \"29cfec31-7adb-41a3-965d-11c1d199b9c3\") " pod="openshift-insights/insights-operator-68d9dd94cf-mqqk9" Jul 14 18:10:29.739625 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.739307 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d052fcc-79b5-45cd-a58e-6bb582e25ab8-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-n4l4d\" (UID: \"5d052fcc-79b5-45cd-a58e-6bb582e25ab8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-n4l4d" Jul 14 18:10:29.739625 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.739599 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-trusted-ca\") pod \"image-registry-5f46bf986b-hh2fk\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:10:29.739685 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.739625 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-7jp42\" (UID: \"1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7jp42" Jul 14 18:10:29.739685 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.739332 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-ca-trust-extracted\") pod \"image-registry-5f46bf986b-hh2fk\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:10:29.739685 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.739673 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-installation-pull-secrets\") pod \"image-registry-5f46bf986b-hh2fk\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:10:29.739855 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.739699 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29cfec31-7adb-41a3-965d-11c1d199b9c3-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-mqqk9\" (UID: \"29cfec31-7adb-41a3-965d-11c1d199b9c3\") " pod="openshift-insights/insights-operator-68d9dd94cf-mqqk9" Jul 14 18:10:29.739855 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.739747 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-stats-auth\") pod \"router-default-5977bb4ff6-wbzmj\" (UID: \"7b2746f9-dc43-4311-aa5b-ad3bc351deb3\") " pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:10:29.739855 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.739782 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-bound-sa-token\") pod \"image-registry-5f46bf986b-hh2fk\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:10:29.739855 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.739804 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/29cfec31-7adb-41a3-965d-11c1d199b9c3-serving-cert\") pod \"insights-operator-68d9dd94cf-mqqk9\" (UID: \"29cfec31-7adb-41a3-965d-11c1d199b9c3\") " pod="openshift-insights/insights-operator-68d9dd94cf-mqqk9" Jul 14 18:10:29.739855 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.739824 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-service-ca-bundle\") pod \"router-default-5977bb4ff6-wbzmj\" (UID: \"7b2746f9-dc43-4311-aa5b-ad3bc351deb3\") " pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:10:29.739855 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.739846 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-metrics-certs\") pod \"router-default-5977bb4ff6-wbzmj\" (UID: \"7b2746f9-dc43-4311-aa5b-ad3bc351deb3\") " pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:10:29.740326 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.739868 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-registry-certificates\") pod \"image-registry-5f46bf986b-hh2fk\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:10:29.740326 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.740022 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6nx8\" (UniqueName: \"kubernetes.io/projected/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-kube-api-access-v6nx8\") pod \"router-default-5977bb4ff6-wbzmj\" (UID: \"7b2746f9-dc43-4311-aa5b-ad3bc351deb3\") " pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:10:29.740574 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.740556 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-trusted-ca\") pod \"image-registry-5f46bf986b-hh2fk\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:10:29.743649 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.743621 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-image-registry-private-configuration\") pod \"image-registry-5f46bf986b-hh2fk\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:10:29.743737 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.743621 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-installation-pull-secrets\") pod \"image-registry-5f46bf986b-hh2fk\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:10:29.748139 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.748120 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jmt26"] Jul 14 18:10:29.748224 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.748202 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-vwp4s" Jul 14 18:10:29.750913 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.750898 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-vghw9\"" Jul 14 18:10:29.752523 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.752504 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dljfz\" (UniqueName: \"kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-kube-api-access-dljfz\") pod \"image-registry-5f46bf986b-hh2fk\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:10:29.752754 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.752733 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-bound-sa-token\") pod \"image-registry-5f46bf986b-hh2fk\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:10:29.762417 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.762402 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zd4lp"] Jul 14 18:10:29.762529 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.762513 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jmt26" Jul 14 18:10:29.765501 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.765484 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 14 18:10:29.765501 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.765496 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hhvn4\"" Jul 14 18:10:29.765618 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.765514 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 14 18:10:29.765618 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.765563 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 14 18:10:29.788065 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.788039 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-vwp4s"] Jul 14 18:10:29.788176 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.788071 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6k9qk"] Jul 14 18:10:29.788176 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.788123 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jmt26"] Jul 14 18:10:29.788176 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.788150 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zd4lp"] Jul 14 18:10:29.788176 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.788152 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zd4lp" Jul 14 18:10:29.788176 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.788169 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-7kfjz"] Jul 14 18:10:29.788386 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.788187 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-lgbnj"] Jul 14 18:10:29.791190 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.791166 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 14 18:10:29.791271 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.791194 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 14 18:10:29.791271 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.791216 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-gwcth\"" Jul 14 18:10:29.791271 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.791217 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 14 18:10:29.791621 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.791603 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 14 18:10:29.840358 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.840327 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49p8l\" (UniqueName: \"kubernetes.io/projected/5d052fcc-79b5-45cd-a58e-6bb582e25ab8-kube-api-access-49p8l\") pod \"cluster-samples-operator-65db858894-n4l4d\" (UID: \"5d052fcc-79b5-45cd-a58e-6bb582e25ab8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-n4l4d" Jul 14 18:10:29.840459 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.840363 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-7jp42\" (UID: \"1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7jp42" Jul 14 18:10:29.840459 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.840381 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44267c25-6aee-4e48-93da-3b222a618eff-metrics-tls\") pod \"dns-default-6k9qk\" (UID: \"44267c25-6aee-4e48-93da-3b222a618eff\") " pod="openshift-dns/dns-default-6k9qk" Jul 14 18:10:29.840459 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.840403 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3f335f2a-958b-49ca-9c11-db21d1694db3-config\") pod \"service-ca-operator-7695fc6c8c-7kfjz\" (UID: \"3f335f2a-958b-49ca-9c11-db21d1694db3\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-7kfjz" Jul 14 18:10:29.840459 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.840424 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29cfec31-7adb-41a3-965d-11c1d199b9c3-tmp\") pod \"insights-operator-68d9dd94cf-mqqk9\" (UID: \"29cfec31-7adb-41a3-965d-11c1d199b9c3\") " pod="openshift-insights/insights-operator-68d9dd94cf-mqqk9" Jul 14 18:10:29.840672 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.840480 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4525dcda-ead6-4800-8d17-58812c289010-config\") pod \"console-operator-7866497494-wqpt2\" (UID: \"4525dcda-ead6-4800-8d17-58812c289010\") " pod="openshift-console-operator/console-operator-7866497494-wqpt2" Jul 14 18:10:29.840672 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:29.840489 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 18:10:29.840672 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.840511 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3dd54969-694e-4b8c-8a8c-21412a2c954e-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-lgbnj\" (UID: \"3dd54969-694e-4b8c-8a8c-21412a2c954e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-lgbnj" Jul 14 18:10:29.840672 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:29.840548 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452-cluster-monitoring-operator-tls podName:1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:30.340531106 +0000 UTC m=+35.177874365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-7jp42" (UID: "1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452") : secret "cluster-monitoring-operator-tls" not found Jul 14 18:10:29.840672 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.840569 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/29cfec31-7adb-41a3-965d-11c1d199b9c3-snapshots\") pod \"insights-operator-68d9dd94cf-mqqk9\" (UID: \"29cfec31-7adb-41a3-965d-11c1d199b9c3\") " pod="openshift-insights/insights-operator-68d9dd94cf-mqqk9" Jul 14 18:10:29.840672 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.840599 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d052fcc-79b5-45cd-a58e-6bb582e25ab8-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-n4l4d\" (UID: \"5d052fcc-79b5-45cd-a58e-6bb582e25ab8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-n4l4d" Jul 14 18:10:29.840923 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:29.840720 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 18:10:29.840923 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.840754 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29cfec31-7adb-41a3-965d-11c1d199b9c3-tmp\") pod \"insights-operator-68d9dd94cf-mqqk9\" (UID: \"29cfec31-7adb-41a3-965d-11c1d199b9c3\") " pod="openshift-insights/insights-operator-68d9dd94cf-mqqk9" Jul 14 18:10:29.840923 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.840775 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h46mg\" (UniqueName: \"kubernetes.io/projected/4525dcda-ead6-4800-8d17-58812c289010-kube-api-access-h46mg\") pod \"console-operator-7866497494-wqpt2\" (UID: \"4525dcda-ead6-4800-8d17-58812c289010\") " pod="openshift-console-operator/console-operator-7866497494-wqpt2" Jul 14 18:10:29.840923 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:29.840780 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d052fcc-79b5-45cd-a58e-6bb582e25ab8-samples-operator-tls podName:5d052fcc-79b5-45cd-a58e-6bb582e25ab8 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:30.340764703 +0000 UTC m=+35.178107957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5d052fcc-79b5-45cd-a58e-6bb582e25ab8-samples-operator-tls") pod "cluster-samples-operator-65db858894-n4l4d" (UID: "5d052fcc-79b5-45cd-a58e-6bb582e25ab8") : secret "samples-operator-tls" not found Jul 14 18:10:29.840923 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.840813 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6nx8\" (UniqueName: \"kubernetes.io/projected/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-kube-api-access-v6nx8\") pod \"router-default-5977bb4ff6-wbzmj\" (UID: \"7b2746f9-dc43-4311-aa5b-ad3bc351deb3\") " pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:10:29.840923 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.840867 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brfrh\" (UniqueName: \"kubernetes.io/projected/3ea371d8-4155-46d3-b0ac-71861adba26d-kube-api-access-brfrh\") pod \"network-check-source-869dcfcdcd-vwp4s\" (UID: \"3ea371d8-4155-46d3-b0ac-71861adba26d\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-vwp4s" Jul 14 18:10:29.840923 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.840913 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-stats-auth\") pod \"router-default-5977bb4ff6-wbzmj\" (UID: \"7b2746f9-dc43-4311-aa5b-ad3bc351deb3\") " pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:10:29.841269 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.840935 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3dd54969-694e-4b8c-8a8c-21412a2c954e-nginx-conf\") pod \"networking-console-plugin-6cf856758c-lgbnj\" (UID: \"3dd54969-694e-4b8c-8a8c-21412a2c954e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-lgbnj" Jul 14 18:10:29.841269 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.841020 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-service-ca-bundle\") pod \"router-default-5977bb4ff6-wbzmj\" (UID: \"7b2746f9-dc43-4311-aa5b-ad3bc351deb3\") " pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:10:29.841269 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.841063 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqkxd\" (UniqueName: \"kubernetes.io/projected/211b531e-f105-44a4-934f-d2e97e981b84-kube-api-access-sqkxd\") pod \"ingress-canary-jmt26\" (UID: \"211b531e-f105-44a4-934f-d2e97e981b84\") " pod="openshift-ingress-canary/ingress-canary-jmt26" Jul 14 18:10:29.841269 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:29.841133 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-service-ca-bundle podName:7b2746f9-dc43-4311-aa5b-ad3bc351deb3 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:30.341121635 +0000 UTC m=+35.178464870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-service-ca-bundle") pod "router-default-5977bb4ff6-wbzmj" (UID: "7b2746f9-dc43-4311-aa5b-ad3bc351deb3") : configmap references non-existent config key: service-ca.crt Jul 14 18:10:29.841269 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.841137 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/29cfec31-7adb-41a3-965d-11c1d199b9c3-snapshots\") pod \"insights-operator-68d9dd94cf-mqqk9\" (UID: \"29cfec31-7adb-41a3-965d-11c1d199b9c3\") " pod="openshift-insights/insights-operator-68d9dd94cf-mqqk9" Jul 14 18:10:29.841269 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.841221 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3f335f2a-958b-49ca-9c11-db21d1694db3-serving-cert\") pod \"service-ca-operator-7695fc6c8c-7kfjz\" (UID: \"3f335f2a-958b-49ca-9c11-db21d1694db3\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-7kfjz" Jul 14 18:10:29.841269 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.841262 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-default-certificate\") pod \"router-default-5977bb4ff6-wbzmj\" (UID: \"7b2746f9-dc43-4311-aa5b-ad3bc351deb3\") " pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:10:29.841585 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.841278 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29cfec31-7adb-41a3-965d-11c1d199b9c3-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-mqqk9\" (UID: \"29cfec31-7adb-41a3-965d-11c1d199b9c3\") " pod="openshift-insights/insights-operator-68d9dd94cf-mqqk9" Jul 14 18:10:29.841585 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.841302 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/211b531e-f105-44a4-934f-d2e97e981b84-cert\") pod \"ingress-canary-jmt26\" (UID: \"211b531e-f105-44a4-934f-d2e97e981b84\") " pod="openshift-ingress-canary/ingress-canary-jmt26" Jul 14 18:10:29.841585 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.841323 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6vfc\" (UniqueName: \"kubernetes.io/projected/938c2572-16f6-4f69-90e7-f33f7e46d4b4-kube-api-access-q6vfc\") pod \"volume-data-source-validator-78495597f6-qldqp\" (UID: \"938c2572-16f6-4f69-90e7-f33f7e46d4b4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-qldqp" Jul 14 18:10:29.841585 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.841340 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4525dcda-ead6-4800-8d17-58812c289010-trusted-ca\") pod \"console-operator-7866497494-wqpt2\" (UID: \"4525dcda-ead6-4800-8d17-58812c289010\") " pod="openshift-console-operator/console-operator-7866497494-wqpt2" Jul 14 18:10:29.841585 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.841366 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44267c25-6aee-4e48-93da-3b222a618eff-tmp-dir\") pod \"dns-default-6k9qk\" (UID: \"44267c25-6aee-4e48-93da-3b222a618eff\") " pod="openshift-dns/dns-default-6k9qk" Jul 14 18:10:29.841585 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.841495 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4525dcda-ead6-4800-8d17-58812c289010-serving-cert\") pod \"console-operator-7866497494-wqpt2\" (UID: \"4525dcda-ead6-4800-8d17-58812c289010\") " pod="openshift-console-operator/console-operator-7866497494-wqpt2" Jul 14 18:10:29.841585 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.841566 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmv72\" (UniqueName: \"kubernetes.io/projected/3f335f2a-958b-49ca-9c11-db21d1694db3-kube-api-access-rmv72\") pod \"service-ca-operator-7695fc6c8c-7kfjz\" (UID: \"3f335f2a-958b-49ca-9c11-db21d1694db3\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-7kfjz" Jul 14 18:10:29.841894 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.841612 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhk7z\" (UniqueName: \"kubernetes.io/projected/29cfec31-7adb-41a3-965d-11c1d199b9c3-kube-api-access-bhk7z\") pod \"insights-operator-68d9dd94cf-mqqk9\" (UID: \"29cfec31-7adb-41a3-965d-11c1d199b9c3\") " pod="openshift-insights/insights-operator-68d9dd94cf-mqqk9" Jul 14 18:10:29.841894 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.841639 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-metrics-certs\") pod \"router-default-5977bb4ff6-wbzmj\" (UID: \"7b2746f9-dc43-4311-aa5b-ad3bc351deb3\") " pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:10:29.841894 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.841679 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zqr7\" (UniqueName: \"kubernetes.io/projected/1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452-kube-api-access-6zqr7\") pod \"cluster-monitoring-operator-5f7f494dd4-7jp42\" (UID: \"1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7jp42" Jul 14 18:10:29.841894 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.841714 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-7jp42\" (UID: \"1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7jp42" Jul 14 18:10:29.841894 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.841739 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29cfec31-7adb-41a3-965d-11c1d199b9c3-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-mqqk9\" (UID: \"29cfec31-7adb-41a3-965d-11c1d199b9c3\") " pod="openshift-insights/insights-operator-68d9dd94cf-mqqk9" Jul 14 18:10:29.841894 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.841768 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q82tv\" (UniqueName: \"kubernetes.io/projected/44267c25-6aee-4e48-93da-3b222a618eff-kube-api-access-q82tv\") pod \"dns-default-6k9qk\" (UID: \"44267c25-6aee-4e48-93da-3b222a618eff\") " pod="openshift-dns/dns-default-6k9qk" Jul 14 18:10:29.841894 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.841795 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/29cfec31-7adb-41a3-965d-11c1d199b9c3-serving-cert\") pod \"insights-operator-68d9dd94cf-mqqk9\" (UID: \"29cfec31-7adb-41a3-965d-11c1d199b9c3\") " pod="openshift-insights/insights-operator-68d9dd94cf-mqqk9" Jul 14 18:10:29.841894 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:29.841779 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 18:10:29.841894 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.841819 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/44267c25-6aee-4e48-93da-3b222a618eff-config-volume\") pod \"dns-default-6k9qk\" (UID: \"44267c25-6aee-4e48-93da-3b222a618eff\") " pod="openshift-dns/dns-default-6k9qk" Jul 14 18:10:29.841894 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:29.841852 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-metrics-certs podName:7b2746f9-dc43-4311-aa5b-ad3bc351deb3 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:30.341838168 +0000 UTC m=+35.179181423 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-metrics-certs") pod "router-default-5977bb4ff6-wbzmj" (UID: "7b2746f9-dc43-4311-aa5b-ad3bc351deb3") : secret "router-metrics-certs-default" not found Jul 14 18:10:29.842370 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.842209 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29cfec31-7adb-41a3-965d-11c1d199b9c3-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-mqqk9\" (UID: \"29cfec31-7adb-41a3-965d-11c1d199b9c3\") " pod="openshift-insights/insights-operator-68d9dd94cf-mqqk9" Jul 14 18:10:29.842370 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.842262 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29cfec31-7adb-41a3-965d-11c1d199b9c3-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-mqqk9\" (UID: \"29cfec31-7adb-41a3-965d-11c1d199b9c3\") " pod="openshift-insights/insights-operator-68d9dd94cf-mqqk9" Jul 14 18:10:29.842931 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.842910 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-7jp42\" (UID: \"1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7jp42" Jul 14 18:10:29.844166 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.844144 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-stats-auth\") pod \"router-default-5977bb4ff6-wbzmj\" (UID: \"7b2746f9-dc43-4311-aa5b-ad3bc351deb3\") " pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:10:29.844257 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.844171 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-default-certificate\") pod \"router-default-5977bb4ff6-wbzmj\" (UID: \"7b2746f9-dc43-4311-aa5b-ad3bc351deb3\") " pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:10:29.844586 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.844565 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/29cfec31-7adb-41a3-965d-11c1d199b9c3-serving-cert\") pod \"insights-operator-68d9dd94cf-mqqk9\" (UID: \"29cfec31-7adb-41a3-965d-11c1d199b9c3\") " pod="openshift-insights/insights-operator-68d9dd94cf-mqqk9" Jul 14 18:10:29.844950 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.844929 2565 generic.go:356] "Generic (PLEG): container finished" podID="42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5" containerID="ea89fee362f2cc215ab5a516f6e8bf50eeb7c78685b84f58d87e15b5a15e0960" exitCode=0 Jul 14 18:10:29.845027 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.844963 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dmt89" event={"ID":"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5","Type":"ContainerDied","Data":"ea89fee362f2cc215ab5a516f6e8bf50eeb7c78685b84f58d87e15b5a15e0960"} Jul 14 18:10:29.853340 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.853241 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6vfc\" (UniqueName: \"kubernetes.io/projected/938c2572-16f6-4f69-90e7-f33f7e46d4b4-kube-api-access-q6vfc\") pod \"volume-data-source-validator-78495597f6-qldqp\" (UID: \"938c2572-16f6-4f69-90e7-f33f7e46d4b4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-qldqp" Jul 14 18:10:29.853498 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.853403 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6nx8\" (UniqueName: \"kubernetes.io/projected/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-kube-api-access-v6nx8\") pod \"router-default-5977bb4ff6-wbzmj\" (UID: \"7b2746f9-dc43-4311-aa5b-ad3bc351deb3\") " pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:10:29.853991 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.853964 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhk7z\" (UniqueName: \"kubernetes.io/projected/29cfec31-7adb-41a3-965d-11c1d199b9c3-kube-api-access-bhk7z\") pod \"insights-operator-68d9dd94cf-mqqk9\" (UID: \"29cfec31-7adb-41a3-965d-11c1d199b9c3\") " pod="openshift-insights/insights-operator-68d9dd94cf-mqqk9" Jul 14 18:10:29.854221 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.854204 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49p8l\" (UniqueName: \"kubernetes.io/projected/5d052fcc-79b5-45cd-a58e-6bb582e25ab8-kube-api-access-49p8l\") pod \"cluster-samples-operator-65db858894-n4l4d\" (UID: \"5d052fcc-79b5-45cd-a58e-6bb582e25ab8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-n4l4d" Jul 14 18:10:29.854883 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.854866 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zqr7\" (UniqueName: \"kubernetes.io/projected/1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452-kube-api-access-6zqr7\") pod \"cluster-monitoring-operator-5f7f494dd4-7jp42\" (UID: \"1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7jp42" Jul 14 18:10:29.882696 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.882674 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-mqqk9" Jul 14 18:10:29.931387 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.931365 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-qldqp" Jul 14 18:10:29.942873 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.942847 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brfrh\" (UniqueName: \"kubernetes.io/projected/3ea371d8-4155-46d3-b0ac-71861adba26d-kube-api-access-brfrh\") pod \"network-check-source-869dcfcdcd-vwp4s\" (UID: \"3ea371d8-4155-46d3-b0ac-71861adba26d\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-vwp4s" Jul 14 18:10:29.942989 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.942911 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3dd54969-694e-4b8c-8a8c-21412a2c954e-nginx-conf\") pod \"networking-console-plugin-6cf856758c-lgbnj\" (UID: \"3dd54969-694e-4b8c-8a8c-21412a2c954e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-lgbnj" Jul 14 18:10:29.942989 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.942955 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqkxd\" (UniqueName: \"kubernetes.io/projected/211b531e-f105-44a4-934f-d2e97e981b84-kube-api-access-sqkxd\") pod \"ingress-canary-jmt26\" (UID: \"211b531e-f105-44a4-934f-d2e97e981b84\") " pod="openshift-ingress-canary/ingress-canary-jmt26" Jul 14 18:10:29.942989 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.942978 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3f335f2a-958b-49ca-9c11-db21d1694db3-serving-cert\") pod \"service-ca-operator-7695fc6c8c-7kfjz\" (UID: \"3f335f2a-958b-49ca-9c11-db21d1694db3\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-7kfjz" Jul 14 18:10:29.943165 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.943005 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2dc349f-af57-4915-b047-20b77acd7ac9-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-zd4lp\" (UID: \"a2dc349f-af57-4915-b047-20b77acd7ac9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zd4lp" Jul 14 18:10:29.943165 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.943120 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/211b531e-f105-44a4-934f-d2e97e981b84-cert\") pod \"ingress-canary-jmt26\" (UID: \"211b531e-f105-44a4-934f-d2e97e981b84\") " pod="openshift-ingress-canary/ingress-canary-jmt26" Jul 14 18:10:29.943268 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.943172 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4525dcda-ead6-4800-8d17-58812c289010-trusted-ca\") pod \"console-operator-7866497494-wqpt2\" (UID: \"4525dcda-ead6-4800-8d17-58812c289010\") " pod="openshift-console-operator/console-operator-7866497494-wqpt2" Jul 14 18:10:29.943268 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.943208 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44267c25-6aee-4e48-93da-3b222a618eff-tmp-dir\") pod \"dns-default-6k9qk\" (UID: \"44267c25-6aee-4e48-93da-3b222a618eff\") " pod="openshift-dns/dns-default-6k9qk" Jul 14 18:10:29.943268 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.943252 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4525dcda-ead6-4800-8d17-58812c289010-serving-cert\") pod \"console-operator-7866497494-wqpt2\" (UID: \"4525dcda-ead6-4800-8d17-58812c289010\") " pod="openshift-console-operator/console-operator-7866497494-wqpt2" Jul 14 18:10:29.943432 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.943326 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmv72\" (UniqueName: \"kubernetes.io/projected/3f335f2a-958b-49ca-9c11-db21d1694db3-kube-api-access-rmv72\") pod \"service-ca-operator-7695fc6c8c-7kfjz\" (UID: \"3f335f2a-958b-49ca-9c11-db21d1694db3\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-7kfjz" Jul 14 18:10:29.943432 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.943412 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q82tv\" (UniqueName: \"kubernetes.io/projected/44267c25-6aee-4e48-93da-3b222a618eff-kube-api-access-q82tv\") pod \"dns-default-6k9qk\" (UID: \"44267c25-6aee-4e48-93da-3b222a618eff\") " pod="openshift-dns/dns-default-6k9qk" Jul 14 18:10:29.943538 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.943445 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/44267c25-6aee-4e48-93da-3b222a618eff-config-volume\") pod \"dns-default-6k9qk\" (UID: \"44267c25-6aee-4e48-93da-3b222a618eff\") " pod="openshift-dns/dns-default-6k9qk" Jul 14 18:10:29.943538 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.943512 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czn57\" (UniqueName: \"kubernetes.io/projected/a2dc349f-af57-4915-b047-20b77acd7ac9-kube-api-access-czn57\") pod \"kube-storage-version-migrator-operator-799cb7f958-zd4lp\" (UID: \"a2dc349f-af57-4915-b047-20b77acd7ac9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zd4lp" Jul 14 18:10:29.943625 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.943556 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44267c25-6aee-4e48-93da-3b222a618eff-metrics-tls\") pod \"dns-default-6k9qk\" (UID: \"44267c25-6aee-4e48-93da-3b222a618eff\") " pod="openshift-dns/dns-default-6k9qk" Jul 14 18:10:29.943625 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.943589 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3f335f2a-958b-49ca-9c11-db21d1694db3-config\") pod \"service-ca-operator-7695fc6c8c-7kfjz\" (UID: \"3f335f2a-958b-49ca-9c11-db21d1694db3\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-7kfjz" Jul 14 18:10:29.943625 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.943607 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44267c25-6aee-4e48-93da-3b222a618eff-tmp-dir\") pod \"dns-default-6k9qk\" (UID: \"44267c25-6aee-4e48-93da-3b222a618eff\") " pod="openshift-dns/dns-default-6k9qk" Jul 14 18:10:29.943748 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.943658 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4525dcda-ead6-4800-8d17-58812c289010-config\") pod \"console-operator-7866497494-wqpt2\" (UID: \"4525dcda-ead6-4800-8d17-58812c289010\") " pod="openshift-console-operator/console-operator-7866497494-wqpt2" Jul 14 18:10:29.943748 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.943681 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3dd54969-694e-4b8c-8a8c-21412a2c954e-nginx-conf\") pod \"networking-console-plugin-6cf856758c-lgbnj\" (UID: \"3dd54969-694e-4b8c-8a8c-21412a2c954e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-lgbnj" Jul 14 18:10:29.943748 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.943689 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3dd54969-694e-4b8c-8a8c-21412a2c954e-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-lgbnj\" (UID: \"3dd54969-694e-4b8c-8a8c-21412a2c954e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-lgbnj" Jul 14 18:10:29.943861 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.943760 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h46mg\" (UniqueName: \"kubernetes.io/projected/4525dcda-ead6-4800-8d17-58812c289010-kube-api-access-h46mg\") pod \"console-operator-7866497494-wqpt2\" (UID: \"4525dcda-ead6-4800-8d17-58812c289010\") " pod="openshift-console-operator/console-operator-7866497494-wqpt2" Jul 14 18:10:29.943861 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:29.943771 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 18:10:29.943861 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.943791 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2dc349f-af57-4915-b047-20b77acd7ac9-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-zd4lp\" (UID: \"a2dc349f-af57-4915-b047-20b77acd7ac9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zd4lp" Jul 14 18:10:29.943861 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:29.943822 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dd54969-694e-4b8c-8a8c-21412a2c954e-networking-console-plugin-cert podName:3dd54969-694e-4b8c-8a8c-21412a2c954e nodeName:}" failed. No retries permitted until 2026-07-14 18:10:30.443805655 +0000 UTC m=+35.281148903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3dd54969-694e-4b8c-8a8c-21412a2c954e-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-lgbnj" (UID: "3dd54969-694e-4b8c-8a8c-21412a2c954e") : secret "networking-console-plugin-cert" not found Jul 14 18:10:29.944707 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.944431 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4525dcda-ead6-4800-8d17-58812c289010-trusted-ca\") pod \"console-operator-7866497494-wqpt2\" (UID: \"4525dcda-ead6-4800-8d17-58812c289010\") " pod="openshift-console-operator/console-operator-7866497494-wqpt2" Jul 14 18:10:29.944707 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.944462 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4525dcda-ead6-4800-8d17-58812c289010-config\") pod \"console-operator-7866497494-wqpt2\" (UID: \"4525dcda-ead6-4800-8d17-58812c289010\") " pod="openshift-console-operator/console-operator-7866497494-wqpt2" Jul 14 18:10:29.944707 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:29.944632 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 18:10:29.944707 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:29.944685 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44267c25-6aee-4e48-93da-3b222a618eff-metrics-tls podName:44267c25-6aee-4e48-93da-3b222a618eff nodeName:}" failed. No retries permitted until 2026-07-14 18:10:30.444669121 +0000 UTC m=+35.282012362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44267c25-6aee-4e48-93da-3b222a618eff-metrics-tls") pod "dns-default-6k9qk" (UID: "44267c25-6aee-4e48-93da-3b222a618eff") : secret "dns-default-metrics-tls" not found Jul 14 18:10:29.945058 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.945035 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3f335f2a-958b-49ca-9c11-db21d1694db3-config\") pod \"service-ca-operator-7695fc6c8c-7kfjz\" (UID: \"3f335f2a-958b-49ca-9c11-db21d1694db3\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-7kfjz" Jul 14 18:10:29.947539 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.947515 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4525dcda-ead6-4800-8d17-58812c289010-serving-cert\") pod \"console-operator-7866497494-wqpt2\" (UID: \"4525dcda-ead6-4800-8d17-58812c289010\") " pod="openshift-console-operator/console-operator-7866497494-wqpt2" Jul 14 18:10:29.947673 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.947599 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/211b531e-f105-44a4-934f-d2e97e981b84-cert\") pod \"ingress-canary-jmt26\" (UID: \"211b531e-f105-44a4-934f-d2e97e981b84\") " pod="openshift-ingress-canary/ingress-canary-jmt26" Jul 14 18:10:29.947745 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.947679 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3f335f2a-958b-49ca-9c11-db21d1694db3-serving-cert\") pod \"service-ca-operator-7695fc6c8c-7kfjz\" (UID: \"3f335f2a-958b-49ca-9c11-db21d1694db3\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-7kfjz" Jul 14 18:10:29.955244 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.955225 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/44267c25-6aee-4e48-93da-3b222a618eff-config-volume\") pod \"dns-default-6k9qk\" (UID: \"44267c25-6aee-4e48-93da-3b222a618eff\") " pod="openshift-dns/dns-default-6k9qk" Jul 14 18:10:29.958679 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.958659 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q82tv\" (UniqueName: \"kubernetes.io/projected/44267c25-6aee-4e48-93da-3b222a618eff-kube-api-access-q82tv\") pod \"dns-default-6k9qk\" (UID: \"44267c25-6aee-4e48-93da-3b222a618eff\") " pod="openshift-dns/dns-default-6k9qk" Jul 14 18:10:29.959073 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.959052 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqkxd\" (UniqueName: \"kubernetes.io/projected/211b531e-f105-44a4-934f-d2e97e981b84-kube-api-access-sqkxd\") pod \"ingress-canary-jmt26\" (UID: \"211b531e-f105-44a4-934f-d2e97e981b84\") " pod="openshift-ingress-canary/ingress-canary-jmt26" Jul 14 18:10:29.959338 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.959317 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h46mg\" (UniqueName: \"kubernetes.io/projected/4525dcda-ead6-4800-8d17-58812c289010-kube-api-access-h46mg\") pod \"console-operator-7866497494-wqpt2\" (UID: \"4525dcda-ead6-4800-8d17-58812c289010\") " pod="openshift-console-operator/console-operator-7866497494-wqpt2" Jul 14 18:10:29.959657 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.959635 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brfrh\" (UniqueName: \"kubernetes.io/projected/3ea371d8-4155-46d3-b0ac-71861adba26d-kube-api-access-brfrh\") pod \"network-check-source-869dcfcdcd-vwp4s\" (UID: \"3ea371d8-4155-46d3-b0ac-71861adba26d\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-vwp4s" Jul 14 18:10:29.960580 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:29.960560 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmv72\" (UniqueName: \"kubernetes.io/projected/3f335f2a-958b-49ca-9c11-db21d1694db3-kube-api-access-rmv72\") pod \"service-ca-operator-7695fc6c8c-7kfjz\" (UID: \"3f335f2a-958b-49ca-9c11-db21d1694db3\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-7kfjz" Jul 14 18:10:30.002671 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.002533 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-wqpt2" Jul 14 18:10:30.027809 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.027641 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-7kfjz" Jul 14 18:10:30.045277 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.045239 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czn57\" (UniqueName: \"kubernetes.io/projected/a2dc349f-af57-4915-b047-20b77acd7ac9-kube-api-access-czn57\") pod \"kube-storage-version-migrator-operator-799cb7f958-zd4lp\" (UID: \"a2dc349f-af57-4915-b047-20b77acd7ac9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zd4lp" Jul 14 18:10:30.045402 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.045346 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2dc349f-af57-4915-b047-20b77acd7ac9-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-zd4lp\" (UID: \"a2dc349f-af57-4915-b047-20b77acd7ac9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zd4lp" Jul 14 18:10:30.045455 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.045426 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2dc349f-af57-4915-b047-20b77acd7ac9-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-zd4lp\" (UID: \"a2dc349f-af57-4915-b047-20b77acd7ac9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zd4lp" Jul 14 18:10:30.046049 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.046021 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2dc349f-af57-4915-b047-20b77acd7ac9-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-zd4lp\" (UID: \"a2dc349f-af57-4915-b047-20b77acd7ac9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zd4lp" Jul 14 18:10:30.048782 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.048762 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2dc349f-af57-4915-b047-20b77acd7ac9-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-zd4lp\" (UID: \"a2dc349f-af57-4915-b047-20b77acd7ac9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zd4lp" Jul 14 18:10:30.055960 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.055902 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czn57\" (UniqueName: \"kubernetes.io/projected/a2dc349f-af57-4915-b047-20b77acd7ac9-kube-api-access-czn57\") pod \"kube-storage-version-migrator-operator-799cb7f958-zd4lp\" (UID: \"a2dc349f-af57-4915-b047-20b77acd7ac9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zd4lp" Jul 14 18:10:30.061696 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.061676 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-vwp4s" Jul 14 18:10:30.069823 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.069802 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jmt26" Jul 14 18:10:30.090585 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.090557 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-mqqk9"] Jul 14 18:10:30.095638 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.095614 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zd4lp" Jul 14 18:10:30.096976 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.096941 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-qldqp"] Jul 14 18:10:30.104063 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:10:30.104031 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938c2572_16f6_4f69_90e7_f33f7e46d4b4.slice/crio-8eb02565829cde364135f4db1fba75900e96e6986e035eabdf5df6de8cf23cea WatchSource:0}: Error finding container 8eb02565829cde364135f4db1fba75900e96e6986e035eabdf5df6de8cf23cea: Status 404 returned error can't find the container with id 8eb02565829cde364135f4db1fba75900e96e6986e035eabdf5df6de8cf23cea Jul 14 18:10:30.247348 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.247168 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-registry-tls\") pod \"image-registry-5f46bf986b-hh2fk\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:10:30.247623 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:30.247563 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 18:10:30.247623 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:30.247580 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f46bf986b-hh2fk: secret "image-registry-tls" not found Jul 14 18:10:30.247824 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:30.247650 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-registry-tls podName:bd03d2f0-2403-414f-b02e-ec3c5e7fb214 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:31.247627196 +0000 UTC m=+36.084970456 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-registry-tls") pod "image-registry-5f46bf986b-hh2fk" (UID: "bd03d2f0-2403-414f-b02e-ec3c5e7fb214") : secret "image-registry-tls" not found Jul 14 18:10:30.292782 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.292749 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-7kfjz"] Jul 14 18:10:30.295919 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:10:30.295884 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f335f2a_958b_49ca_9c11_db21d1694db3.slice/crio-4dbc0f3bbfd7fcce2446ef7532f1fa3f3c74dc0627011cb384bd07767869e584 WatchSource:0}: Error finding container 4dbc0f3bbfd7fcce2446ef7532f1fa3f3c74dc0627011cb384bd07767869e584: Status 404 returned error can't find the container with id 4dbc0f3bbfd7fcce2446ef7532f1fa3f3c74dc0627011cb384bd07767869e584 Jul 14 18:10:30.297792 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.297761 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jmt26"] Jul 14 18:10:30.300616 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:10:30.300513 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod211b531e_f105_44a4_934f_d2e97e981b84.slice/crio-e99cd402806ee452241fe689bb3cd815b8ea987393ed8101e98a409ee14cc540 WatchSource:0}: Error finding container e99cd402806ee452241fe689bb3cd815b8ea987393ed8101e98a409ee14cc540: Status 404 returned error can't find the container with id e99cd402806ee452241fe689bb3cd815b8ea987393ed8101e98a409ee14cc540 Jul 14 18:10:30.306012 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.305994 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-vwp4s"] Jul 14 18:10:30.310491 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.310468 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zd4lp"] Jul 14 18:10:30.313864 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:10:30.313842 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2dc349f_af57_4915_b047_20b77acd7ac9.slice/crio-d489cc03da222832df508a0e6e6e40291f1c2bd68662b42a874adea3df4e042f WatchSource:0}: Error finding container d489cc03da222832df508a0e6e6e40291f1c2bd68662b42a874adea3df4e042f: Status 404 returned error can't find the container with id d489cc03da222832df508a0e6e6e40291f1c2bd68662b42a874adea3df4e042f Jul 14 18:10:30.315997 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.315979 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-wqpt2"] Jul 14 18:10:30.318579 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:10:30.318550 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4525dcda_ead6_4800_8d17_58812c289010.slice/crio-62eca97afd716369bac0b4b8853d046c74c2ec8cd32c93824946e5e13677b595 WatchSource:0}: Error finding container 62eca97afd716369bac0b4b8853d046c74c2ec8cd32c93824946e5e13677b595: Status 404 returned error can't find the container with id 62eca97afd716369bac0b4b8853d046c74c2ec8cd32c93824946e5e13677b595 Jul 14 18:10:30.348011 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.347989 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d052fcc-79b5-45cd-a58e-6bb582e25ab8-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-n4l4d\" (UID: \"5d052fcc-79b5-45cd-a58e-6bb582e25ab8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-n4l4d" Jul 14 18:10:30.348337 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.348039 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-service-ca-bundle\") pod \"router-default-5977bb4ff6-wbzmj\" (UID: \"7b2746f9-dc43-4311-aa5b-ad3bc351deb3\") " pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:10:30.348337 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.348088 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-metrics-certs\") pod \"router-default-5977bb4ff6-wbzmj\" (UID: \"7b2746f9-dc43-4311-aa5b-ad3bc351deb3\") " pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:10:30.348337 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:30.348143 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 18:10:30.348337 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:30.348200 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 18:10:30.348337 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:30.348214 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d052fcc-79b5-45cd-a58e-6bb582e25ab8-samples-operator-tls podName:5d052fcc-79b5-45cd-a58e-6bb582e25ab8 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:31.348185687 +0000 UTC m=+36.185528922 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5d052fcc-79b5-45cd-a58e-6bb582e25ab8-samples-operator-tls") pod "cluster-samples-operator-65db858894-n4l4d" (UID: "5d052fcc-79b5-45cd-a58e-6bb582e25ab8") : secret "samples-operator-tls" not found Jul 14 18:10:30.348337 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.348147 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-7jp42\" (UID: \"1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7jp42" Jul 14 18:10:30.348337 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:30.348228 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 18:10:30.348337 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:30.348234 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-service-ca-bundle podName:7b2746f9-dc43-4311-aa5b-ad3bc351deb3 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:31.348224674 +0000 UTC m=+36.185567911 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-service-ca-bundle") pod "router-default-5977bb4ff6-wbzmj" (UID: "7b2746f9-dc43-4311-aa5b-ad3bc351deb3") : configmap references non-existent config key: service-ca.crt Jul 14 18:10:30.348337 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:30.348267 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452-cluster-monitoring-operator-tls podName:1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:31.348259513 +0000 UTC m=+36.185602752 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-7jp42" (UID: "1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452") : secret "cluster-monitoring-operator-tls" not found Jul 14 18:10:30.348337 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:30.348333 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-metrics-certs podName:7b2746f9-dc43-4311-aa5b-ad3bc351deb3 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:31.348324796 +0000 UTC m=+36.185668063 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-metrics-certs") pod "router-default-5977bb4ff6-wbzmj" (UID: "7b2746f9-dc43-4311-aa5b-ad3bc351deb3") : secret "router-metrics-certs-default" not found Jul 14 18:10:30.449184 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.449121 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3dd54969-694e-4b8c-8a8c-21412a2c954e-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-lgbnj\" (UID: \"3dd54969-694e-4b8c-8a8c-21412a2c954e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-lgbnj" Jul 14 18:10:30.449287 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:30.449262 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 18:10:30.449323 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.449300 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44267c25-6aee-4e48-93da-3b222a618eff-metrics-tls\") pod \"dns-default-6k9qk\" (UID: \"44267c25-6aee-4e48-93da-3b222a618eff\") " pod="openshift-dns/dns-default-6k9qk" Jul 14 18:10:30.449323 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:30.449319 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dd54969-694e-4b8c-8a8c-21412a2c954e-networking-console-plugin-cert podName:3dd54969-694e-4b8c-8a8c-21412a2c954e nodeName:}" failed. No retries permitted until 2026-07-14 18:10:31.449302516 +0000 UTC m=+36.286645766 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3dd54969-694e-4b8c-8a8c-21412a2c954e-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-lgbnj" (UID: "3dd54969-694e-4b8c-8a8c-21412a2c954e") : secret "networking-console-plugin-cert" not found Jul 14 18:10:30.449388 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:30.449361 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 18:10:30.449415 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:30.449394 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44267c25-6aee-4e48-93da-3b222a618eff-metrics-tls podName:44267c25-6aee-4e48-93da-3b222a618eff nodeName:}" failed. No retries permitted until 2026-07-14 18:10:31.44938531 +0000 UTC m=+36.286728545 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44267c25-6aee-4e48-93da-3b222a618eff-metrics-tls") pod "dns-default-6k9qk" (UID: "44267c25-6aee-4e48-93da-3b222a618eff") : secret "dns-default-metrics-tls" not found Jul 14 18:10:30.848985 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.848951 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-qldqp" event={"ID":"938c2572-16f6-4f69-90e7-f33f7e46d4b4","Type":"ContainerStarted","Data":"8eb02565829cde364135f4db1fba75900e96e6986e035eabdf5df6de8cf23cea"} Jul 14 18:10:30.851886 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.851862 2565 generic.go:356] "Generic (PLEG): container finished" podID="42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5" containerID="95dbdcfacd1d0d7d5c64782129f6571e1096b61bf9c0189a66b4df071bd665e5" exitCode=0 Jul 14 18:10:30.852001 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.851918 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dmt89" event={"ID":"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5","Type":"ContainerDied","Data":"95dbdcfacd1d0d7d5c64782129f6571e1096b61bf9c0189a66b4df071bd665e5"} Jul 14 18:10:30.853141 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.853120 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-vwp4s" event={"ID":"3ea371d8-4155-46d3-b0ac-71861adba26d","Type":"ContainerStarted","Data":"392034de50f4af13f3ffcbc283631ec921813ca2320dd6f727e1254311b865c5"} Jul 14 18:10:30.854144 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.854125 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-7kfjz" event={"ID":"3f335f2a-958b-49ca-9c11-db21d1694db3","Type":"ContainerStarted","Data":"4dbc0f3bbfd7fcce2446ef7532f1fa3f3c74dc0627011cb384bd07767869e584"} Jul 14 18:10:30.855027 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.855006 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-mqqk9" event={"ID":"29cfec31-7adb-41a3-965d-11c1d199b9c3","Type":"ContainerStarted","Data":"63610ae818c07c383a3332d042d835956c928dfd8992209d70b919428cbe95aa"} Jul 14 18:10:30.855978 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.855938 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zd4lp" event={"ID":"a2dc349f-af57-4915-b047-20b77acd7ac9","Type":"ContainerStarted","Data":"d489cc03da222832df508a0e6e6e40291f1c2bd68662b42a874adea3df4e042f"} Jul 14 18:10:30.856947 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.856925 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jmt26" event={"ID":"211b531e-f105-44a4-934f-d2e97e981b84","Type":"ContainerStarted","Data":"e99cd402806ee452241fe689bb3cd815b8ea987393ed8101e98a409ee14cc540"} Jul 14 18:10:30.857781 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:30.857766 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-wqpt2" event={"ID":"4525dcda-ead6-4800-8d17-58812c289010","Type":"ContainerStarted","Data":"62eca97afd716369bac0b4b8853d046c74c2ec8cd32c93824946e5e13677b595"} Jul 14 18:10:31.257198 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:31.257116 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-registry-tls\") pod \"image-registry-5f46bf986b-hh2fk\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:10:31.257360 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:31.257327 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 18:10:31.257360 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:31.257351 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f46bf986b-hh2fk: secret "image-registry-tls" not found Jul 14 18:10:31.257490 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:31.257416 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-registry-tls podName:bd03d2f0-2403-414f-b02e-ec3c5e7fb214 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:33.257400721 +0000 UTC m=+38.094743956 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-registry-tls") pod "image-registry-5f46bf986b-hh2fk" (UID: "bd03d2f0-2403-414f-b02e-ec3c5e7fb214") : secret "image-registry-tls" not found Jul 14 18:10:31.358279 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:31.358246 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-7jp42\" (UID: \"1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7jp42" Jul 14 18:10:31.358973 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:31.358326 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d052fcc-79b5-45cd-a58e-6bb582e25ab8-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-n4l4d\" (UID: \"5d052fcc-79b5-45cd-a58e-6bb582e25ab8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-n4l4d" Jul 14 18:10:31.358973 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:31.358383 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-service-ca-bundle\") pod \"router-default-5977bb4ff6-wbzmj\" (UID: \"7b2746f9-dc43-4311-aa5b-ad3bc351deb3\") " pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:10:31.358973 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:31.358414 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 18:10:31.358973 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:31.358438 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-metrics-certs\") pod \"router-default-5977bb4ff6-wbzmj\" (UID: \"7b2746f9-dc43-4311-aa5b-ad3bc351deb3\") " pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:10:31.358973 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:31.358474 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 18:10:31.358973 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:31.358488 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452-cluster-monitoring-operator-tls podName:1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:33.358465506 +0000 UTC m=+38.195808746 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-7jp42" (UID: "1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452") : secret "cluster-monitoring-operator-tls" not found Jul 14 18:10:31.358973 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:31.358537 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d052fcc-79b5-45cd-a58e-6bb582e25ab8-samples-operator-tls podName:5d052fcc-79b5-45cd-a58e-6bb582e25ab8 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:33.358516627 +0000 UTC m=+38.195859864 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5d052fcc-79b5-45cd-a58e-6bb582e25ab8-samples-operator-tls") pod "cluster-samples-operator-65db858894-n4l4d" (UID: "5d052fcc-79b5-45cd-a58e-6bb582e25ab8") : secret "samples-operator-tls" not found Jul 14 18:10:31.358973 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:31.358557 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 18:10:31.358973 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:31.358624 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-metrics-certs podName:7b2746f9-dc43-4311-aa5b-ad3bc351deb3 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:33.358594578 +0000 UTC m=+38.195937837 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-metrics-certs") pod "router-default-5977bb4ff6-wbzmj" (UID: "7b2746f9-dc43-4311-aa5b-ad3bc351deb3") : secret "router-metrics-certs-default" not found Jul 14 18:10:31.358973 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:31.358657 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-service-ca-bundle podName:7b2746f9-dc43-4311-aa5b-ad3bc351deb3 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:33.35864364 +0000 UTC m=+38.195986877 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-service-ca-bundle") pod "router-default-5977bb4ff6-wbzmj" (UID: "7b2746f9-dc43-4311-aa5b-ad3bc351deb3") : configmap references non-existent config key: service-ca.crt Jul 14 18:10:31.459746 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:31.459698 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44267c25-6aee-4e48-93da-3b222a618eff-metrics-tls\") pod \"dns-default-6k9qk\" (UID: \"44267c25-6aee-4e48-93da-3b222a618eff\") " pod="openshift-dns/dns-default-6k9qk" Jul 14 18:10:31.459920 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:31.459772 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3dd54969-694e-4b8c-8a8c-21412a2c954e-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-lgbnj\" (UID: \"3dd54969-694e-4b8c-8a8c-21412a2c954e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-lgbnj" Jul 14 18:10:31.460000 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:31.459983 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 18:10:31.460063 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:31.460050 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dd54969-694e-4b8c-8a8c-21412a2c954e-networking-console-plugin-cert podName:3dd54969-694e-4b8c-8a8c-21412a2c954e nodeName:}" failed. No retries permitted until 2026-07-14 18:10:33.460032651 +0000 UTC m=+38.297375887 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3dd54969-694e-4b8c-8a8c-21412a2c954e-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-lgbnj" (UID: "3dd54969-694e-4b8c-8a8c-21412a2c954e") : secret "networking-console-plugin-cert" not found Jul 14 18:10:31.460499 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:31.460471 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 18:10:31.460590 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:31.460540 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44267c25-6aee-4e48-93da-3b222a618eff-metrics-tls podName:44267c25-6aee-4e48-93da-3b222a618eff nodeName:}" failed. No retries permitted until 2026-07-14 18:10:33.460524322 +0000 UTC m=+38.297867562 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44267c25-6aee-4e48-93da-3b222a618eff-metrics-tls") pod "dns-default-6k9qk" (UID: "44267c25-6aee-4e48-93da-3b222a618eff") : secret "dns-default-metrics-tls" not found Jul 14 18:10:31.868954 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:31.868123 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dmt89" event={"ID":"42e4c55d-de1f-4bda-9f4b-f1dcf46eb2b5","Type":"ContainerStarted","Data":"e50847d54871883008e4a44d20d6aa2ec0066c59d808f27e7266250843577c54"} Jul 14 18:10:31.906740 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:31.906676 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dmt89" podStartSLOduration=8.135193907 podStartE2EDuration="36.906653293s" totalImagesPullingTime="28.771459386s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:09:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:31.905444267 +0000 UTC m=+36.742787525" watchObservedRunningTime="2026-07-14 18:10:31.906653293 +0000 UTC m=+36.743996552" Jul 14 18:10:33.278459 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:33.278430 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-registry-tls\") pod \"image-registry-5f46bf986b-hh2fk\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:10:33.278942 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:33.278626 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 18:10:33.278942 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:33.278646 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f46bf986b-hh2fk: secret "image-registry-tls" not found Jul 14 18:10:33.278942 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:33.278713 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-registry-tls podName:bd03d2f0-2403-414f-b02e-ec3c5e7fb214 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:37.278694338 +0000 UTC m=+42.116037592 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-registry-tls") pod "image-registry-5f46bf986b-hh2fk" (UID: "bd03d2f0-2403-414f-b02e-ec3c5e7fb214") : secret "image-registry-tls" not found Jul 14 18:10:33.380262 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:33.379454 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d052fcc-79b5-45cd-a58e-6bb582e25ab8-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-n4l4d\" (UID: \"5d052fcc-79b5-45cd-a58e-6bb582e25ab8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-n4l4d" Jul 14 18:10:33.380262 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:33.379539 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-service-ca-bundle\") pod \"router-default-5977bb4ff6-wbzmj\" (UID: \"7b2746f9-dc43-4311-aa5b-ad3bc351deb3\") " pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:10:33.380262 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:33.379597 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-metrics-certs\") pod \"router-default-5977bb4ff6-wbzmj\" (UID: \"7b2746f9-dc43-4311-aa5b-ad3bc351deb3\") " pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:10:33.380262 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:33.379652 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-7jp42\" (UID: \"1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7jp42" Jul 14 18:10:33.380262 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:33.379787 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 18:10:33.380262 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:33.379866 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452-cluster-monitoring-operator-tls podName:1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:37.379848885 +0000 UTC m=+42.217192147 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-7jp42" (UID: "1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452") : secret "cluster-monitoring-operator-tls" not found Jul 14 18:10:33.380262 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:33.380227 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-service-ca-bundle podName:7b2746f9-dc43-4311-aa5b-ad3bc351deb3 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:37.380211192 +0000 UTC m=+42.217554442 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-service-ca-bundle") pod "router-default-5977bb4ff6-wbzmj" (UID: "7b2746f9-dc43-4311-aa5b-ad3bc351deb3") : configmap references non-existent config key: service-ca.crt Jul 14 18:10:33.380262 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:33.380241 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 18:10:33.380262 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:33.380248 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 18:10:33.380676 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:33.380292 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d052fcc-79b5-45cd-a58e-6bb582e25ab8-samples-operator-tls podName:5d052fcc-79b5-45cd-a58e-6bb582e25ab8 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:37.380279288 +0000 UTC m=+42.217622557 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5d052fcc-79b5-45cd-a58e-6bb582e25ab8-samples-operator-tls") pod "cluster-samples-operator-65db858894-n4l4d" (UID: "5d052fcc-79b5-45cd-a58e-6bb582e25ab8") : secret "samples-operator-tls" not found Jul 14 18:10:33.380676 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:33.380312 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-metrics-certs podName:7b2746f9-dc43-4311-aa5b-ad3bc351deb3 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:37.3802987 +0000 UTC m=+42.217641937 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-metrics-certs") pod "router-default-5977bb4ff6-wbzmj" (UID: "7b2746f9-dc43-4311-aa5b-ad3bc351deb3") : secret "router-metrics-certs-default" not found Jul 14 18:10:33.480626 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:33.480594 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44267c25-6aee-4e48-93da-3b222a618eff-metrics-tls\") pod \"dns-default-6k9qk\" (UID: \"44267c25-6aee-4e48-93da-3b222a618eff\") " pod="openshift-dns/dns-default-6k9qk" Jul 14 18:10:33.480808 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:33.480650 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3dd54969-694e-4b8c-8a8c-21412a2c954e-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-lgbnj\" (UID: \"3dd54969-694e-4b8c-8a8c-21412a2c954e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-lgbnj" Jul 14 18:10:33.480871 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:33.480809 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 18:10:33.480923 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:33.480872 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dd54969-694e-4b8c-8a8c-21412a2c954e-networking-console-plugin-cert podName:3dd54969-694e-4b8c-8a8c-21412a2c954e nodeName:}" failed. No retries permitted until 2026-07-14 18:10:37.48085226 +0000 UTC m=+42.318195499 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3dd54969-694e-4b8c-8a8c-21412a2c954e-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-lgbnj" (UID: "3dd54969-694e-4b8c-8a8c-21412a2c954e") : secret "networking-console-plugin-cert" not found Jul 14 18:10:33.481208 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:33.481091 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 18:10:33.481208 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:33.481193 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44267c25-6aee-4e48-93da-3b222a618eff-metrics-tls podName:44267c25-6aee-4e48-93da-3b222a618eff nodeName:}" failed. No retries permitted until 2026-07-14 18:10:37.481174506 +0000 UTC m=+42.318517749 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44267c25-6aee-4e48-93da-3b222a618eff-metrics-tls") pod "dns-default-6k9qk" (UID: "44267c25-6aee-4e48-93da-3b222a618eff") : secret "dns-default-metrics-tls" not found Jul 14 18:10:37.317351 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:37.317306 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-registry-tls\") pod \"image-registry-5f46bf986b-hh2fk\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:10:37.317874 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:37.317502 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 18:10:37.317874 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:37.317522 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f46bf986b-hh2fk: secret "image-registry-tls" not found Jul 14 18:10:37.317874 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:37.317579 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-registry-tls podName:bd03d2f0-2403-414f-b02e-ec3c5e7fb214 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:45.317563429 +0000 UTC m=+50.154906668 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-registry-tls") pod "image-registry-5f46bf986b-hh2fk" (UID: "bd03d2f0-2403-414f-b02e-ec3c5e7fb214") : secret "image-registry-tls" not found Jul 14 18:10:37.418447 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:37.418413 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-service-ca-bundle\") pod \"router-default-5977bb4ff6-wbzmj\" (UID: \"7b2746f9-dc43-4311-aa5b-ad3bc351deb3\") " pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:10:37.418657 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:37.418487 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-metrics-certs\") pod \"router-default-5977bb4ff6-wbzmj\" (UID: \"7b2746f9-dc43-4311-aa5b-ad3bc351deb3\") " pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:10:37.418657 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:37.418546 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-7jp42\" (UID: \"1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7jp42" Jul 14 18:10:37.418657 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:37.418624 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-service-ca-bundle podName:7b2746f9-dc43-4311-aa5b-ad3bc351deb3 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:45.41858374 +0000 UTC m=+50.255926991 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-service-ca-bundle") pod "router-default-5977bb4ff6-wbzmj" (UID: "7b2746f9-dc43-4311-aa5b-ad3bc351deb3") : configmap references non-existent config key: service-ca.crt Jul 14 18:10:37.418836 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:37.418663 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d052fcc-79b5-45cd-a58e-6bb582e25ab8-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-n4l4d\" (UID: \"5d052fcc-79b5-45cd-a58e-6bb582e25ab8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-n4l4d" Jul 14 18:10:37.418836 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:37.418674 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 18:10:37.418836 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:37.418680 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 18:10:37.418836 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:37.418714 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452-cluster-monitoring-operator-tls podName:1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:45.418702974 +0000 UTC m=+50.256046227 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-7jp42" (UID: "1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452") : secret "cluster-monitoring-operator-tls" not found Jul 14 18:10:37.418836 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:37.418748 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-metrics-certs podName:7b2746f9-dc43-4311-aa5b-ad3bc351deb3 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:45.418730463 +0000 UTC m=+50.256073701 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-metrics-certs") pod "router-default-5977bb4ff6-wbzmj" (UID: "7b2746f9-dc43-4311-aa5b-ad3bc351deb3") : secret "router-metrics-certs-default" not found Jul 14 18:10:37.418836 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:37.418789 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 18:10:37.419069 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:37.418870 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d052fcc-79b5-45cd-a58e-6bb582e25ab8-samples-operator-tls podName:5d052fcc-79b5-45cd-a58e-6bb582e25ab8 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:45.418852069 +0000 UTC m=+50.256195321 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5d052fcc-79b5-45cd-a58e-6bb582e25ab8-samples-operator-tls") pod "cluster-samples-operator-65db858894-n4l4d" (UID: "5d052fcc-79b5-45cd-a58e-6bb582e25ab8") : secret "samples-operator-tls" not found Jul 14 18:10:37.519747 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:37.519690 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3dd54969-694e-4b8c-8a8c-21412a2c954e-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-lgbnj\" (UID: \"3dd54969-694e-4b8c-8a8c-21412a2c954e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-lgbnj" Jul 14 18:10:37.519884 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:37.519833 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 18:10:37.519884 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:37.519841 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44267c25-6aee-4e48-93da-3b222a618eff-metrics-tls\") pod \"dns-default-6k9qk\" (UID: \"44267c25-6aee-4e48-93da-3b222a618eff\") " pod="openshift-dns/dns-default-6k9qk" Jul 14 18:10:37.519998 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:37.519889 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dd54969-694e-4b8c-8a8c-21412a2c954e-networking-console-plugin-cert podName:3dd54969-694e-4b8c-8a8c-21412a2c954e nodeName:}" failed. No retries permitted until 2026-07-14 18:10:45.519875613 +0000 UTC m=+50.357218849 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3dd54969-694e-4b8c-8a8c-21412a2c954e-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-lgbnj" (UID: "3dd54969-694e-4b8c-8a8c-21412a2c954e") : secret "networking-console-plugin-cert" not found Jul 14 18:10:37.519998 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:37.519930 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 18:10:37.519998 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:37.519971 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44267c25-6aee-4e48-93da-3b222a618eff-metrics-tls podName:44267c25-6aee-4e48-93da-3b222a618eff nodeName:}" failed. No retries permitted until 2026-07-14 18:10:45.519960947 +0000 UTC m=+50.357304182 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44267c25-6aee-4e48-93da-3b222a618eff-metrics-tls") pod "dns-default-6k9qk" (UID: "44267c25-6aee-4e48-93da-3b222a618eff") : secret "dns-default-metrics-tls" not found Jul 14 18:10:38.888587 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:38.888536 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-qldqp" event={"ID":"938c2572-16f6-4f69-90e7-f33f7e46d4b4","Type":"ContainerStarted","Data":"ad519d4cdf439785a51e6b90820725fd57e16483569d40e2beb7692335a7f9ee"} Jul 14 18:10:38.890731 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:38.890347 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-vwp4s" event={"ID":"3ea371d8-4155-46d3-b0ac-71861adba26d","Type":"ContainerStarted","Data":"d4421d4c17094ff228114125a7da9c38445e781a4b759c25e867768bd068bcee"} Jul 14 18:10:38.892000 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:38.891939 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-7kfjz" event={"ID":"3f335f2a-958b-49ca-9c11-db21d1694db3","Type":"ContainerStarted","Data":"6fcf88800a523eca2a11e8010d2997175bc6f437b36fd7fdfaa40e47f908a887"} Jul 14 18:10:38.893583 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:38.893543 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-mqqk9" event={"ID":"29cfec31-7adb-41a3-965d-11c1d199b9c3","Type":"ContainerStarted","Data":"175debad6ded35d41fe5ff7266f7c9c82917cd7f86cb713e2e69b2e7ff00a122"} Jul 14 18:10:38.895018 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:38.894988 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zd4lp" event={"ID":"a2dc349f-af57-4915-b047-20b77acd7ac9","Type":"ContainerStarted","Data":"4362c8e05ab6a4a1306e86cbd626b2def7279fa5548cf040ca0c0414ed5cebdb"} Jul 14 18:10:38.896559 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:38.896530 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jmt26" event={"ID":"211b531e-f105-44a4-934f-d2e97e981b84","Type":"ContainerStarted","Data":"1f3c17e7661878a17fe5c9fec53944cdb128685af10ae932c58329c560c5cc84"} Jul 14 18:10:38.898027 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:38.898010 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wqpt2_4525dcda-ead6-4800-8d17-58812c289010/console-operator/0.log" Jul 14 18:10:38.898125 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:38.898046 2565 generic.go:356] "Generic (PLEG): container finished" podID="4525dcda-ead6-4800-8d17-58812c289010" containerID="dd8d915e5e91c4b218a7045b1cfed88a8a8231123fdb428f0e2e774835475607" exitCode=255 Jul 14 18:10:38.898125 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:38.898072 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-wqpt2" event={"ID":"4525dcda-ead6-4800-8d17-58812c289010","Type":"ContainerDied","Data":"dd8d915e5e91c4b218a7045b1cfed88a8a8231123fdb428f0e2e774835475607"} Jul 14 18:10:38.898261 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:38.898247 2565 scope.go:117] "RemoveContainer" containerID="dd8d915e5e91c4b218a7045b1cfed88a8a8231123fdb428f0e2e774835475607" Jul 14 18:10:38.912751 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:38.908514 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-qldqp" podStartSLOduration=28.026080756 podStartE2EDuration="34.908498546s" totalImagesPullingTime="6.88241779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:10:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:38.90604062 +0000 UTC m=+43.743383877" watchObservedRunningTime="2026-07-14 18:10:38.908498546 +0000 UTC m=+43.745841803" Jul 14 18:10:38.954645 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:38.954579 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-7kfjz" podStartSLOduration=21.675476976 podStartE2EDuration="28.954551342s" totalImagesPullingTime="7.279074366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:38.953451748 +0000 UTC m=+43.790795030" watchObservedRunningTime="2026-07-14 18:10:38.954551342 +0000 UTC m=+43.791894601" Jul 14 18:10:39.007450 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:39.006918 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-vwp4s" podStartSLOduration=22.378818087 podStartE2EDuration="30.006894656s" totalImagesPullingTime="7.628076569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:10:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:39.005289217 +0000 UTC m=+43.842632477" watchObservedRunningTime="2026-07-14 18:10:39.006894656 +0000 UTC m=+43.844237917" Jul 14 18:10:39.027655 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:39.027569 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jmt26" podStartSLOduration=2.33468463 podStartE2EDuration="10.027547463s" totalImagesPullingTime="7.692862833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:10:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:39.026019845 +0000 UTC m=+43.863363102" watchObservedRunningTime="2026-07-14 18:10:39.027547463 +0000 UTC m=+43.864890722" Jul 14 18:10:39.059555 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:39.057838 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-mqqk9" podStartSLOduration=27.585758161 podStartE2EDuration="35.057817866s" totalImagesPullingTime="7.472059705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:10:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:39.056970526 +0000 UTC m=+43.894313781" watchObservedRunningTime="2026-07-14 18:10:39.057817866 +0000 UTC m=+43.895161124" Jul 14 18:10:39.087193 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:39.087145 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zd4lp" podStartSLOduration=23.825647486 podStartE2EDuration="31.087123442s" totalImagesPullingTime="7.261475956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:39.085332377 +0000 UTC m=+43.922675817" watchObservedRunningTime="2026-07-14 18:10:39.087123442 +0000 UTC m=+43.924466698" Jul 14 18:10:39.649957 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:39.649925 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9g9f7"] Jul 14 18:10:39.661779 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:39.661756 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9g9f7" Jul 14 18:10:39.664446 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:39.664419 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 14 18:10:39.664554 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:39.664456 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 14 18:10:39.664605 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:39.664557 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-266xb\"" Jul 14 18:10:39.670625 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:39.670583 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9g9f7"] Jul 14 18:10:39.739034 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:39.738738 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9c5e5426-5ad9-4ecc-93aa-577db1551181-crio-socket\") pod \"insights-runtime-extractor-9g9f7\" (UID: \"9c5e5426-5ad9-4ecc-93aa-577db1551181\") " pod="openshift-insights/insights-runtime-extractor-9g9f7" Jul 14 18:10:39.739034 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:39.738784 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzffw\" (UniqueName: \"kubernetes.io/projected/9c5e5426-5ad9-4ecc-93aa-577db1551181-kube-api-access-rzffw\") pod \"insights-runtime-extractor-9g9f7\" (UID: \"9c5e5426-5ad9-4ecc-93aa-577db1551181\") " pod="openshift-insights/insights-runtime-extractor-9g9f7" Jul 14 18:10:39.739034 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:39.738856 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c5e5426-5ad9-4ecc-93aa-577db1551181-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9g9f7\" (UID: \"9c5e5426-5ad9-4ecc-93aa-577db1551181\") " pod="openshift-insights/insights-runtime-extractor-9g9f7" Jul 14 18:10:39.739034 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:39.738917 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c5e5426-5ad9-4ecc-93aa-577db1551181-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9g9f7\" (UID: \"9c5e5426-5ad9-4ecc-93aa-577db1551181\") " pod="openshift-insights/insights-runtime-extractor-9g9f7" Jul 14 18:10:39.739034 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:39.738988 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c5e5426-5ad9-4ecc-93aa-577db1551181-data-volume\") pod \"insights-runtime-extractor-9g9f7\" (UID: \"9c5e5426-5ad9-4ecc-93aa-577db1551181\") " pod="openshift-insights/insights-runtime-extractor-9g9f7" Jul 14 18:10:39.840339 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:39.840309 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9c5e5426-5ad9-4ecc-93aa-577db1551181-crio-socket\") pod \"insights-runtime-extractor-9g9f7\" (UID: \"9c5e5426-5ad9-4ecc-93aa-577db1551181\") " pod="openshift-insights/insights-runtime-extractor-9g9f7" Jul 14 18:10:39.840339 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:39.840340 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzffw\" (UniqueName: \"kubernetes.io/projected/9c5e5426-5ad9-4ecc-93aa-577db1551181-kube-api-access-rzffw\") pod \"insights-runtime-extractor-9g9f7\" (UID: \"9c5e5426-5ad9-4ecc-93aa-577db1551181\") " pod="openshift-insights/insights-runtime-extractor-9g9f7" Jul 14 18:10:39.840584 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:39.840391 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c5e5426-5ad9-4ecc-93aa-577db1551181-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9g9f7\" (UID: \"9c5e5426-5ad9-4ecc-93aa-577db1551181\") " pod="openshift-insights/insights-runtime-extractor-9g9f7" Jul 14 18:10:39.840584 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:39.840454 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9c5e5426-5ad9-4ecc-93aa-577db1551181-crio-socket\") pod \"insights-runtime-extractor-9g9f7\" (UID: \"9c5e5426-5ad9-4ecc-93aa-577db1551181\") " pod="openshift-insights/insights-runtime-extractor-9g9f7" Jul 14 18:10:39.840584 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:39.840474 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 18:10:39.840584 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:39.840545 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c5e5426-5ad9-4ecc-93aa-577db1551181-insights-runtime-extractor-tls podName:9c5e5426-5ad9-4ecc-93aa-577db1551181 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:40.340521686 +0000 UTC m=+45.177864928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c5e5426-5ad9-4ecc-93aa-577db1551181-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9g9f7" (UID: "9c5e5426-5ad9-4ecc-93aa-577db1551181") : secret "insights-runtime-extractor-tls" not found Jul 14 18:10:39.840791 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:39.840606 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c5e5426-5ad9-4ecc-93aa-577db1551181-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9g9f7\" (UID: \"9c5e5426-5ad9-4ecc-93aa-577db1551181\") " pod="openshift-insights/insights-runtime-extractor-9g9f7" Jul 14 18:10:39.840791 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:39.840688 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c5e5426-5ad9-4ecc-93aa-577db1551181-data-volume\") pod \"insights-runtime-extractor-9g9f7\" (UID: \"9c5e5426-5ad9-4ecc-93aa-577db1551181\") " pod="openshift-insights/insights-runtime-extractor-9g9f7" Jul 14 18:10:39.840791 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:39.840727 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 18:10:39.840791 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:39.840779 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c5e5426-5ad9-4ecc-93aa-577db1551181-kube-rbac-proxy-cm podName:9c5e5426-5ad9-4ecc-93aa-577db1551181 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:40.340761154 +0000 UTC m=+45.178104391 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c5e5426-5ad9-4ecc-93aa-577db1551181-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9g9f7" (UID: "9c5e5426-5ad9-4ecc-93aa-577db1551181") : configmap "kube-rbac-proxy" not found Jul 14 18:10:39.840999 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:39.840973 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c5e5426-5ad9-4ecc-93aa-577db1551181-data-volume\") pod \"insights-runtime-extractor-9g9f7\" (UID: \"9c5e5426-5ad9-4ecc-93aa-577db1551181\") " pod="openshift-insights/insights-runtime-extractor-9g9f7" Jul 14 18:10:39.852483 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:39.852464 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzffw\" (UniqueName: \"kubernetes.io/projected/9c5e5426-5ad9-4ecc-93aa-577db1551181-kube-api-access-rzffw\") pod \"insights-runtime-extractor-9g9f7\" (UID: \"9c5e5426-5ad9-4ecc-93aa-577db1551181\") " pod="openshift-insights/insights-runtime-extractor-9g9f7" Jul 14 18:10:39.904941 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:39.904890 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wqpt2_4525dcda-ead6-4800-8d17-58812c289010/console-operator/1.log" Jul 14 18:10:39.905350 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:39.905323 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wqpt2_4525dcda-ead6-4800-8d17-58812c289010/console-operator/0.log" Jul 14 18:10:39.905400 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:39.905353 2565 generic.go:356] "Generic (PLEG): container finished" podID="4525dcda-ead6-4800-8d17-58812c289010" containerID="8146d9cf9d07a88ba1bf9f77b552bbb4b380fecafbb4658ec956c492a93dd05d" exitCode=255 Jul 14 18:10:39.905443 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:39.905433 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-wqpt2" event={"ID":"4525dcda-ead6-4800-8d17-58812c289010","Type":"ContainerDied","Data":"8146d9cf9d07a88ba1bf9f77b552bbb4b380fecafbb4658ec956c492a93dd05d"} Jul 14 18:10:39.905489 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:39.905481 2565 scope.go:117] "RemoveContainer" containerID="dd8d915e5e91c4b218a7045b1cfed88a8a8231123fdb428f0e2e774835475607" Jul 14 18:10:39.906035 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:39.905671 2565 scope.go:117] "RemoveContainer" containerID="8146d9cf9d07a88ba1bf9f77b552bbb4b380fecafbb4658ec956c492a93dd05d" Jul 14 18:10:39.906035 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:39.905843 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-wqpt2_openshift-console-operator(4525dcda-ead6-4800-8d17-58812c289010)\"" pod="openshift-console-operator/console-operator-7866497494-wqpt2" podUID="4525dcda-ead6-4800-8d17-58812c289010" Jul 14 18:10:40.003287 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:40.003251 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-wqpt2" Jul 14 18:10:40.003427 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:40.003307 2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-wqpt2" Jul 14 18:10:40.349711 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:40.346501 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c5e5426-5ad9-4ecc-93aa-577db1551181-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9g9f7\" (UID: \"9c5e5426-5ad9-4ecc-93aa-577db1551181\") " pod="openshift-insights/insights-runtime-extractor-9g9f7" Jul 14 18:10:40.349711 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:40.346663 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 18:10:40.349711 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:40.346699 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c5e5426-5ad9-4ecc-93aa-577db1551181-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9g9f7\" (UID: \"9c5e5426-5ad9-4ecc-93aa-577db1551181\") " pod="openshift-insights/insights-runtime-extractor-9g9f7" Jul 14 18:10:40.349711 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:40.346766 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c5e5426-5ad9-4ecc-93aa-577db1551181-insights-runtime-extractor-tls podName:9c5e5426-5ad9-4ecc-93aa-577db1551181 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:41.346744911 +0000 UTC m=+46.184088167 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c5e5426-5ad9-4ecc-93aa-577db1551181-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9g9f7" (UID: "9c5e5426-5ad9-4ecc-93aa-577db1551181") : secret "insights-runtime-extractor-tls" not found Jul 14 18:10:40.349711 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:40.346815 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 18:10:40.349711 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:40.346885 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c5e5426-5ad9-4ecc-93aa-577db1551181-kube-rbac-proxy-cm podName:9c5e5426-5ad9-4ecc-93aa-577db1551181 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:41.346868464 +0000 UTC m=+46.184211703 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c5e5426-5ad9-4ecc-93aa-577db1551181-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9g9f7" (UID: "9c5e5426-5ad9-4ecc-93aa-577db1551181") : configmap "kube-rbac-proxy" not found Jul 14 18:10:40.910424 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:40.910397 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wqpt2_4525dcda-ead6-4800-8d17-58812c289010/console-operator/1.log" Jul 14 18:10:40.910927 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:40.910820 2565 scope.go:117] "RemoveContainer" containerID="8146d9cf9d07a88ba1bf9f77b552bbb4b380fecafbb4658ec956c492a93dd05d" Jul 14 18:10:40.911042 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:40.911019 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-wqpt2_openshift-console-operator(4525dcda-ead6-4800-8d17-58812c289010)\"" pod="openshift-console-operator/console-operator-7866497494-wqpt2" podUID="4525dcda-ead6-4800-8d17-58812c289010" Jul 14 18:10:41.355858 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:41.355830 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c5e5426-5ad9-4ecc-93aa-577db1551181-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9g9f7\" (UID: \"9c5e5426-5ad9-4ecc-93aa-577db1551181\") " pod="openshift-insights/insights-runtime-extractor-9g9f7" Jul 14 18:10:41.356043 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:41.355910 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c5e5426-5ad9-4ecc-93aa-577db1551181-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9g9f7\" (UID: \"9c5e5426-5ad9-4ecc-93aa-577db1551181\") " pod="openshift-insights/insights-runtime-extractor-9g9f7" Jul 14 18:10:41.356043 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:41.355966 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 18:10:41.356043 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:41.356023 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 18:10:41.356043 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:41.356031 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c5e5426-5ad9-4ecc-93aa-577db1551181-insights-runtime-extractor-tls podName:9c5e5426-5ad9-4ecc-93aa-577db1551181 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:43.356013914 +0000 UTC m=+48.193357150 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c5e5426-5ad9-4ecc-93aa-577db1551181-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9g9f7" (UID: "9c5e5426-5ad9-4ecc-93aa-577db1551181") : secret "insights-runtime-extractor-tls" not found Jul 14 18:10:41.356240 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:41.356083 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c5e5426-5ad9-4ecc-93aa-577db1551181-kube-rbac-proxy-cm podName:9c5e5426-5ad9-4ecc-93aa-577db1551181 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:43.356070156 +0000 UTC m=+48.193413396 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c5e5426-5ad9-4ecc-93aa-577db1551181-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9g9f7" (UID: "9c5e5426-5ad9-4ecc-93aa-577db1551181") : configmap "kube-rbac-proxy" not found Jul 14 18:10:41.913438 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:41.913410 2565 scope.go:117] "RemoveContainer" containerID="8146d9cf9d07a88ba1bf9f77b552bbb4b380fecafbb4658ec956c492a93dd05d" Jul 14 18:10:41.913788 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:41.913564 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-wqpt2_openshift-console-operator(4525dcda-ead6-4800-8d17-58812c289010)\"" pod="openshift-console-operator/console-operator-7866497494-wqpt2" podUID="4525dcda-ead6-4800-8d17-58812c289010" Jul 14 18:10:42.045168 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:42.045137 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ds4bd_0089b7f3-1d21-4a3d-b5d8-fc1cd289f06e/dns-node-resolver/0.log" Jul 14 18:10:43.037556 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:43.037530 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qx9gz_87ca2cc2-58d4-4899-8bc1-2b5bd93ebe6f/node-ca/0.log" Jul 14 18:10:43.373601 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:43.373512 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c5e5426-5ad9-4ecc-93aa-577db1551181-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9g9f7\" (UID: \"9c5e5426-5ad9-4ecc-93aa-577db1551181\") " pod="openshift-insights/insights-runtime-extractor-9g9f7" Jul 14 18:10:43.373601 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:43.373566 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c5e5426-5ad9-4ecc-93aa-577db1551181-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9g9f7\" (UID: \"9c5e5426-5ad9-4ecc-93aa-577db1551181\") " pod="openshift-insights/insights-runtime-extractor-9g9f7" Jul 14 18:10:43.373778 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:43.373653 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 18:10:43.373778 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:43.373675 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 18:10:43.373778 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:43.373709 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c5e5426-5ad9-4ecc-93aa-577db1551181-kube-rbac-proxy-cm podName:9c5e5426-5ad9-4ecc-93aa-577db1551181 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:47.373696209 +0000 UTC m=+52.211039444 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c5e5426-5ad9-4ecc-93aa-577db1551181-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9g9f7" (UID: "9c5e5426-5ad9-4ecc-93aa-577db1551181") : configmap "kube-rbac-proxy" not found Jul 14 18:10:43.373778 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:43.373732 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c5e5426-5ad9-4ecc-93aa-577db1551181-insights-runtime-extractor-tls podName:9c5e5426-5ad9-4ecc-93aa-577db1551181 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:47.373717155 +0000 UTC m=+52.211060390 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c5e5426-5ad9-4ecc-93aa-577db1551181-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9g9f7" (UID: "9c5e5426-5ad9-4ecc-93aa-577db1551181") : secret "insights-runtime-extractor-tls" not found Jul 14 18:10:43.638292 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:43.638216 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jmt26_211b531e-f105-44a4-934f-d2e97e981b84/serve-healthcheck-canary/0.log" Jul 14 18:10:44.437222 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:44.437190 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-zd4lp_a2dc349f-af57-4915-b047-20b77acd7ac9/kube-storage-version-migrator-operator/0.log" Jul 14 18:10:45.393291 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:45.393257 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-registry-tls\") pod \"image-registry-5f46bf986b-hh2fk\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:10:45.393461 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:45.393404 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 18:10:45.393461 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:45.393423 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f46bf986b-hh2fk: secret "image-registry-tls" not found Jul 14 18:10:45.393530 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:45.393478 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-registry-tls podName:bd03d2f0-2403-414f-b02e-ec3c5e7fb214 nodeName:}" failed. No retries permitted until 2026-07-14 18:11:01.393461047 +0000 UTC m=+66.230804298 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-registry-tls") pod "image-registry-5f46bf986b-hh2fk" (UID: "bd03d2f0-2403-414f-b02e-ec3c5e7fb214") : secret "image-registry-tls" not found Jul 14 18:10:45.494140 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:45.494111 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-metrics-certs\") pod \"router-default-5977bb4ff6-wbzmj\" (UID: \"7b2746f9-dc43-4311-aa5b-ad3bc351deb3\") " pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:10:45.494485 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:45.494188 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-7jp42\" (UID: \"1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7jp42" Jul 14 18:10:45.494485 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:45.494194 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 18:10:45.494485 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:45.494240 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d052fcc-79b5-45cd-a58e-6bb582e25ab8-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-n4l4d\" (UID: \"5d052fcc-79b5-45cd-a58e-6bb582e25ab8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-n4l4d" Jul 14 18:10:45.494485 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:45.494250 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-metrics-certs podName:7b2746f9-dc43-4311-aa5b-ad3bc351deb3 nodeName:}" failed. No retries permitted until 2026-07-14 18:11:01.494236519 +0000 UTC m=+66.331579755 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-metrics-certs") pod "router-default-5977bb4ff6-wbzmj" (UID: "7b2746f9-dc43-4311-aa5b-ad3bc351deb3") : secret "router-metrics-certs-default" not found Jul 14 18:10:45.494485 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:45.494314 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-service-ca-bundle\") pod \"router-default-5977bb4ff6-wbzmj\" (UID: \"7b2746f9-dc43-4311-aa5b-ad3bc351deb3\") " pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:10:45.494485 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:45.494324 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 18:10:45.494485 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:45.494324 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 18:10:45.494485 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:45.494370 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d052fcc-79b5-45cd-a58e-6bb582e25ab8-samples-operator-tls podName:5d052fcc-79b5-45cd-a58e-6bb582e25ab8 nodeName:}" failed. No retries permitted until 2026-07-14 18:11:01.494356449 +0000 UTC m=+66.331699708 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5d052fcc-79b5-45cd-a58e-6bb582e25ab8-samples-operator-tls") pod "cluster-samples-operator-65db858894-n4l4d" (UID: "5d052fcc-79b5-45cd-a58e-6bb582e25ab8") : secret "samples-operator-tls" not found Jul 14 18:10:45.494485 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:45.494384 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452-cluster-monitoring-operator-tls podName:1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452 nodeName:}" failed. No retries permitted until 2026-07-14 18:11:01.494378291 +0000 UTC m=+66.331721526 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-7jp42" (UID: "1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452") : secret "cluster-monitoring-operator-tls" not found Jul 14 18:10:45.494485 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:45.494410 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-service-ca-bundle podName:7b2746f9-dc43-4311-aa5b-ad3bc351deb3 nodeName:}" failed. No retries permitted until 2026-07-14 18:11:01.49440287 +0000 UTC m=+66.331746105 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-service-ca-bundle") pod "router-default-5977bb4ff6-wbzmj" (UID: "7b2746f9-dc43-4311-aa5b-ad3bc351deb3") : configmap references non-existent config key: service-ca.crt Jul 14 18:10:45.594946 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:45.594905 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44267c25-6aee-4e48-93da-3b222a618eff-metrics-tls\") pod \"dns-default-6k9qk\" (UID: \"44267c25-6aee-4e48-93da-3b222a618eff\") " pod="openshift-dns/dns-default-6k9qk" Jul 14 18:10:45.595174 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:45.594959 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3dd54969-694e-4b8c-8a8c-21412a2c954e-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-lgbnj\" (UID: \"3dd54969-694e-4b8c-8a8c-21412a2c954e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-lgbnj" Jul 14 18:10:45.595174 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:45.595079 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 18:10:45.595174 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:45.595090 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 18:10:45.595174 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:45.595173 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dd54969-694e-4b8c-8a8c-21412a2c954e-networking-console-plugin-cert podName:3dd54969-694e-4b8c-8a8c-21412a2c954e nodeName:}" failed. No retries permitted until 2026-07-14 18:11:01.595155894 +0000 UTC m=+66.432499129 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3dd54969-694e-4b8c-8a8c-21412a2c954e-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-lgbnj" (UID: "3dd54969-694e-4b8c-8a8c-21412a2c954e") : secret "networking-console-plugin-cert" not found Jul 14 18:10:45.595409 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:45.595191 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44267c25-6aee-4e48-93da-3b222a618eff-metrics-tls podName:44267c25-6aee-4e48-93da-3b222a618eff nodeName:}" failed. No retries permitted until 2026-07-14 18:11:01.595184509 +0000 UTC m=+66.432527746 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44267c25-6aee-4e48-93da-3b222a618eff-metrics-tls") pod "dns-default-6k9qk" (UID: "44267c25-6aee-4e48-93da-3b222a618eff") : secret "dns-default-metrics-tls" not found Jul 14 18:10:47.409805 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:47.409775 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c5e5426-5ad9-4ecc-93aa-577db1551181-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9g9f7\" (UID: \"9c5e5426-5ad9-4ecc-93aa-577db1551181\") " pod="openshift-insights/insights-runtime-extractor-9g9f7" Jul 14 18:10:47.410175 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:47.409905 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 18:10:47.410175 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:47.409936 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c5e5426-5ad9-4ecc-93aa-577db1551181-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9g9f7\" (UID: \"9c5e5426-5ad9-4ecc-93aa-577db1551181\") " pod="openshift-insights/insights-runtime-extractor-9g9f7" Jul 14 18:10:47.410175 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:47.409970 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c5e5426-5ad9-4ecc-93aa-577db1551181-kube-rbac-proxy-cm podName:9c5e5426-5ad9-4ecc-93aa-577db1551181 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:55.409952211 +0000 UTC m=+60.247295448 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c5e5426-5ad9-4ecc-93aa-577db1551181-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9g9f7" (UID: "9c5e5426-5ad9-4ecc-93aa-577db1551181") : configmap "kube-rbac-proxy" not found Jul 14 18:10:47.410175 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:47.410028 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 18:10:47.410175 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:47.410070 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c5e5426-5ad9-4ecc-93aa-577db1551181-insights-runtime-extractor-tls podName:9c5e5426-5ad9-4ecc-93aa-577db1551181 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:55.410058471 +0000 UTC m=+60.247401706 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c5e5426-5ad9-4ecc-93aa-577db1551181-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9g9f7" (UID: "9c5e5426-5ad9-4ecc-93aa-577db1551181") : secret "insights-runtime-extractor-tls" not found Jul 14 18:10:55.481291 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:55.481253 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c5e5426-5ad9-4ecc-93aa-577db1551181-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9g9f7\" (UID: \"9c5e5426-5ad9-4ecc-93aa-577db1551181\") " pod="openshift-insights/insights-runtime-extractor-9g9f7" Jul 14 18:10:55.481770 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:55.481322 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c5e5426-5ad9-4ecc-93aa-577db1551181-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9g9f7\" (UID: \"9c5e5426-5ad9-4ecc-93aa-577db1551181\") " pod="openshift-insights/insights-runtime-extractor-9g9f7" Jul 14 18:10:55.481770 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:55.481419 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 18:10:55.481770 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:10:55.481483 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c5e5426-5ad9-4ecc-93aa-577db1551181-insights-runtime-extractor-tls podName:9c5e5426-5ad9-4ecc-93aa-577db1551181 nodeName:}" failed. No retries permitted until 2026-07-14 18:11:11.481467494 +0000 UTC m=+76.318810729 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c5e5426-5ad9-4ecc-93aa-577db1551181-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9g9f7" (UID: "9c5e5426-5ad9-4ecc-93aa-577db1551181") : secret "insights-runtime-extractor-tls" not found Jul 14 18:10:55.481876 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:55.481840 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c5e5426-5ad9-4ecc-93aa-577db1551181-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9g9f7\" (UID: \"9c5e5426-5ad9-4ecc-93aa-577db1551181\") " pod="openshift-insights/insights-runtime-extractor-9g9f7" Jul 14 18:10:55.685588 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:55.685560 2565 scope.go:117] "RemoveContainer" containerID="8146d9cf9d07a88ba1bf9f77b552bbb4b380fecafbb4658ec956c492a93dd05d" Jul 14 18:10:55.844284 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:55.844258 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f99bz" Jul 14 18:10:55.956830 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:55.956803 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wqpt2_4525dcda-ead6-4800-8d17-58812c289010/console-operator/1.log" Jul 14 18:10:55.956988 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:55.956866 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-wqpt2" event={"ID":"4525dcda-ead6-4800-8d17-58812c289010","Type":"ContainerStarted","Data":"3d88568e56af237e36ea53204d7d9684d701cee2becd2b07785b9f3471764b09"} Jul 14 18:10:55.957143 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:55.957126 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-wqpt2" Jul 14 18:10:55.984445 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:55.984401 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-wqpt2" podStartSLOduration=44.301723075 podStartE2EDuration="51.984386524s" totalImagesPullingTime="7.682663449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:10:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:55.983410739 +0000 UTC m=+60.820753994" watchObservedRunningTime="2026-07-14 18:10:55.984386524 +0000 UTC m=+60.821729781" Jul 14 18:10:56.212825 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:10:56.212792 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-wqpt2" Jul 14 18:11:01.428910 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.428878 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/015d11c8-2009-4c76-815a-c2d87c42e7b5-original-pull-secret\") pod \"global-pull-secret-syncer-6czgz\" (UID: \"015d11c8-2009-4c76-815a-c2d87c42e7b5\") " pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:11:01.429374 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.428933 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-registry-tls\") pod \"image-registry-5f46bf986b-hh2fk\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:11:01.431513 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.431491 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-registry-tls\") pod \"image-registry-5f46bf986b-hh2fk\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:11:01.431966 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.431935 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 18:11:01.441475 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.441455 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/015d11c8-2009-4c76-815a-c2d87c42e7b5-original-pull-secret\") pod \"global-pull-secret-syncer-6czgz\" (UID: \"015d11c8-2009-4c76-815a-c2d87c42e7b5\") " pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:11:01.523334 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.523305 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6czgz" Jul 14 18:11:01.529645 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.529627 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-service-ca-bundle\") pod \"router-default-5977bb4ff6-wbzmj\" (UID: \"7b2746f9-dc43-4311-aa5b-ad3bc351deb3\") " pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:11:01.529713 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.529668 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-metrics-certs\") pod \"router-default-5977bb4ff6-wbzmj\" (UID: \"7b2746f9-dc43-4311-aa5b-ad3bc351deb3\") " pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:11:01.529713 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.529700 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1fc5b88-ec77-4f28-83ff-82725cf2a5e1-metrics-certs\") pod \"network-metrics-daemon-rbgtx\" (UID: \"a1fc5b88-ec77-4f28-83ff-82725cf2a5e1\") " pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:11:01.529822 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.529731 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-7jp42\" (UID: \"1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7jp42" Jul 14 18:11:01.529822 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.529805 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d052fcc-79b5-45cd-a58e-6bb582e25ab8-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-n4l4d\" (UID: \"5d052fcc-79b5-45cd-a58e-6bb582e25ab8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-n4l4d" Jul 14 18:11:01.530287 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.530263 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-service-ca-bundle\") pod \"router-default-5977bb4ff6-wbzmj\" (UID: \"7b2746f9-dc43-4311-aa5b-ad3bc351deb3\") " pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:11:01.532172 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.532146 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b2746f9-dc43-4311-aa5b-ad3bc351deb3-metrics-certs\") pod \"router-default-5977bb4ff6-wbzmj\" (UID: \"7b2746f9-dc43-4311-aa5b-ad3bc351deb3\") " pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:11:01.532265 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.532187 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-7jp42\" (UID: \"1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7jp42" Jul 14 18:11:01.532265 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.532214 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d052fcc-79b5-45cd-a58e-6bb582e25ab8-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-n4l4d\" (UID: \"5d052fcc-79b5-45cd-a58e-6bb582e25ab8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-n4l4d" Jul 14 18:11:01.533240 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.533217 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 18:11:01.541699 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.541682 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1fc5b88-ec77-4f28-83ff-82725cf2a5e1-metrics-certs\") pod \"network-metrics-daemon-rbgtx\" (UID: \"a1fc5b88-ec77-4f28-83ff-82725cf2a5e1\") " pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:11:01.630494 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.630461 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44267c25-6aee-4e48-93da-3b222a618eff-metrics-tls\") pod \"dns-default-6k9qk\" (UID: \"44267c25-6aee-4e48-93da-3b222a618eff\") " pod="openshift-dns/dns-default-6k9qk" Jul 14 18:11:01.630638 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.630510 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3dd54969-694e-4b8c-8a8c-21412a2c954e-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-lgbnj\" (UID: \"3dd54969-694e-4b8c-8a8c-21412a2c954e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-lgbnj" Jul 14 18:11:01.630638 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.630550 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhhw6\" (UniqueName: \"kubernetes.io/projected/65462b89-c7cc-4843-a6e9-baa2aac800af-kube-api-access-vhhw6\") pod \"network-check-target-x92qm\" (UID: \"65462b89-c7cc-4843-a6e9-baa2aac800af\") " pod="openshift-network-diagnostics/network-check-target-x92qm" Jul 14 18:11:01.632875 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.632854 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44267c25-6aee-4e48-93da-3b222a618eff-metrics-tls\") pod \"dns-default-6k9qk\" (UID: \"44267c25-6aee-4e48-93da-3b222a618eff\") " pod="openshift-dns/dns-default-6k9qk" Jul 14 18:11:01.633900 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.633881 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhhw6\" (UniqueName: \"kubernetes.io/projected/65462b89-c7cc-4843-a6e9-baa2aac800af-kube-api-access-vhhw6\") pod \"network-check-target-x92qm\" (UID: \"65462b89-c7cc-4843-a6e9-baa2aac800af\") " pod="openshift-network-diagnostics/network-check-target-x92qm" Jul 14 18:11:01.634159 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.634136 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3dd54969-694e-4b8c-8a8c-21412a2c954e-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-lgbnj\" (UID: \"3dd54969-694e-4b8c-8a8c-21412a2c954e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-lgbnj" Jul 14 18:11:01.658420 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.658398 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kfgwx\"" Jul 14 18:11:01.666195 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.666174 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:11:01.690883 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.690829 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6czgz"] Jul 14 18:11:01.693884 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:11:01.693858 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod015d11c8_2009_4c76_815a_c2d87c42e7b5.slice/crio-ce6e6754455195bc11344ba737de372e45256d70552f62c68621c909faaed5f3 WatchSource:0}: Error finding container ce6e6754455195bc11344ba737de372e45256d70552f62c68621c909faaed5f3: Status 404 returned error can't find the container with id ce6e6754455195bc11344ba737de372e45256d70552f62c68621c909faaed5f3 Jul 14 18:11:01.708236 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.708213 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-n6z24\"" Jul 14 18:11:01.715800 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.715780 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7jp42" Jul 14 18:11:01.750459 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.750215 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-lk8cm\"" Jul 14 18:11:01.756023 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.755773 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-n4l4d" Jul 14 18:11:01.775160 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.774823 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-np2nj\"" Jul 14 18:11:01.780561 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.780540 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:11:01.788093 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.788066 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zklb7\"" Jul 14 18:11:01.795169 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.795143 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6k9qk" Jul 14 18:11:01.812841 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.812424 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jkhfz\"" Jul 14 18:11:01.817958 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.817934 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5vgxh\"" Jul 14 18:11:01.819274 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.819249 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbgtx" Jul 14 18:11:01.824738 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.824166 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x92qm" Jul 14 18:11:01.841752 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.841599 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-92j4h\"" Jul 14 18:11:01.848340 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.848309 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-lgbnj" Jul 14 18:11:01.863674 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.863036 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f46bf986b-hh2fk"] Jul 14 18:11:01.869714 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:11:01.869662 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd03d2f0_2403_414f_b02e_ec3c5e7fb214.slice/crio-f0258f2cd3206ff0ce1fbae5fe12f7b6a8d351885dabdb76293dcb1bf1dad4c0 WatchSource:0}: Error finding container f0258f2cd3206ff0ce1fbae5fe12f7b6a8d351885dabdb76293dcb1bf1dad4c0: Status 404 returned error can't find the container with id f0258f2cd3206ff0ce1fbae5fe12f7b6a8d351885dabdb76293dcb1bf1dad4c0 Jul 14 18:11:01.895703 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.895583 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7jp42"] Jul 14 18:11:01.967780 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.967690 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-n4l4d"] Jul 14 18:11:01.983380 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.981225 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" event={"ID":"bd03d2f0-2403-414f-b02e-ec3c5e7fb214","Type":"ContainerStarted","Data":"f0258f2cd3206ff0ce1fbae5fe12f7b6a8d351885dabdb76293dcb1bf1dad4c0"} Jul 14 18:11:01.983380 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.982863 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7jp42" event={"ID":"1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452","Type":"ContainerStarted","Data":"aa9f625531dd87fdb8b73f4a8854f60d309e7805d7abb31464094269770dccb5"} Jul 14 18:11:01.985022 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:01.984919 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6czgz" event={"ID":"015d11c8-2009-4c76-815a-c2d87c42e7b5","Type":"ContainerStarted","Data":"ce6e6754455195bc11344ba737de372e45256d70552f62c68621c909faaed5f3"} Jul 14 18:11:02.016403 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:02.016014 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5977bb4ff6-wbzmj"] Jul 14 18:11:02.017837 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:11:02.017793 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b2746f9_dc43_4311_aa5b_ad3bc351deb3.slice/crio-11bc9374d615ae92e75f66398a6026487ef57e4bf6e17e8d2a7e73652c04cc5a WatchSource:0}: Error finding container 11bc9374d615ae92e75f66398a6026487ef57e4bf6e17e8d2a7e73652c04cc5a: Status 404 returned error can't find the container with id 11bc9374d615ae92e75f66398a6026487ef57e4bf6e17e8d2a7e73652c04cc5a Jul 14 18:11:02.031377 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:02.031352 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6k9qk"] Jul 14 18:11:02.065130 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:02.065026 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x92qm"] Jul 14 18:11:02.069957 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:02.069933 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rbgtx"] Jul 14 18:11:02.077158 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:11:02.077130 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65462b89_c7cc_4843_a6e9_baa2aac800af.slice/crio-2725a178f8c3181d08d6276ade221c0030af56ba7e2d16679b2638597ff397a5 WatchSource:0}: Error finding container 2725a178f8c3181d08d6276ade221c0030af56ba7e2d16679b2638597ff397a5: Status 404 returned error can't find the container with id 2725a178f8c3181d08d6276ade221c0030af56ba7e2d16679b2638597ff397a5 Jul 14 18:11:02.077390 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:11:02.077374 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1fc5b88_ec77_4f28_83ff_82725cf2a5e1.slice/crio-1e4c3a138508dfd85bb50cff67a6e2866172cc140595ccdb925ffee1c9e8d5a8 WatchSource:0}: Error finding container 1e4c3a138508dfd85bb50cff67a6e2866172cc140595ccdb925ffee1c9e8d5a8: Status 404 returned error can't find the container with id 1e4c3a138508dfd85bb50cff67a6e2866172cc140595ccdb925ffee1c9e8d5a8 Jul 14 18:11:02.087566 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:02.086313 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-lgbnj"] Jul 14 18:11:02.092696 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:11:02.090779 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dd54969_694e_4b8c_8a8c_21412a2c954e.slice/crio-bebabedef712a35e2df467c2b0ee680cc032497c22a08f917cbc2c24520ff52d WatchSource:0}: Error finding container bebabedef712a35e2df467c2b0ee680cc032497c22a08f917cbc2c24520ff52d: Status 404 returned error can't find the container with id bebabedef712a35e2df467c2b0ee680cc032497c22a08f917cbc2c24520ff52d Jul 14 18:11:02.990731 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:02.990695 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rbgtx" event={"ID":"a1fc5b88-ec77-4f28-83ff-82725cf2a5e1","Type":"ContainerStarted","Data":"1e4c3a138508dfd85bb50cff67a6e2866172cc140595ccdb925ffee1c9e8d5a8"} Jul 14 18:11:02.993617 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:02.993585 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" event={"ID":"bd03d2f0-2403-414f-b02e-ec3c5e7fb214","Type":"ContainerStarted","Data":"92d2306188f1833d505a7cda6506114a9fe16699fed5b21be9e179ba1e3244bb"} Jul 14 18:11:02.994334 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:02.994315 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:11:02.996309 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:02.996148 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-lgbnj" event={"ID":"3dd54969-694e-4b8c-8a8c-21412a2c954e","Type":"ContainerStarted","Data":"bebabedef712a35e2df467c2b0ee680cc032497c22a08f917cbc2c24520ff52d"} Jul 14 18:11:02.997697 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:02.997647 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-n4l4d" event={"ID":"5d052fcc-79b5-45cd-a58e-6bb582e25ab8","Type":"ContainerStarted","Data":"22dbae9308378502523a1cb6f187bb2e2cf5a221d3f77003404510f84a90e92f"} Jul 14 18:11:02.999528 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:02.999496 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x92qm" event={"ID":"65462b89-c7cc-4843-a6e9-baa2aac800af","Type":"ContainerStarted","Data":"4a5c14417b6d7666085a2d5c6cafbba5b02518544c4442d891ae992a0a84d39d"} Jul 14 18:11:02.999629 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:02.999544 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x92qm" event={"ID":"65462b89-c7cc-4843-a6e9-baa2aac800af","Type":"ContainerStarted","Data":"2725a178f8c3181d08d6276ade221c0030af56ba7e2d16679b2638597ff397a5"} Jul 14 18:11:02.999694 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:02.999659 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-x92qm" Jul 14 18:11:03.001512 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.001486 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6k9qk" event={"ID":"44267c25-6aee-4e48-93da-3b222a618eff","Type":"ContainerStarted","Data":"349969d837721497f82d2fefcc614bb48d976399a54d0d0252caf7f1bf3d585a"} Jul 14 18:11:03.003986 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.003962 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" event={"ID":"7b2746f9-dc43-4311-aa5b-ad3bc351deb3","Type":"ContainerStarted","Data":"dcb676ea0cceb7ac2d99a41c371f728b22ae0308952fc32cdf3acd81b3f5ca8c"} Jul 14 18:11:03.004087 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.004005 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" event={"ID":"7b2746f9-dc43-4311-aa5b-ad3bc351deb3","Type":"ContainerStarted","Data":"11bc9374d615ae92e75f66398a6026487ef57e4bf6e17e8d2a7e73652c04cc5a"} Jul 14 18:11:03.029011 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.028458 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" podStartSLOduration=67.028438536 podStartE2EDuration="1m7.028438536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:03.026522295 +0000 UTC m=+67.863865552" watchObservedRunningTime="2026-07-14 18:11:03.028438536 +0000 UTC m=+67.865781796" Jul 14 18:11:03.055180 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.055129 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" podStartSLOduration=59.055084441 podStartE2EDuration="59.055084441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:03.054182349 +0000 UTC m=+67.891525608" watchObservedRunningTime="2026-07-14 18:11:03.055084441 +0000 UTC m=+67.892427696" Jul 14 18:11:03.078800 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.077348 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-x92qm" podStartSLOduration=67.077334916 podStartE2EDuration="1m7.077334916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:03.076883613 +0000 UTC m=+67.914226871" watchObservedRunningTime="2026-07-14 18:11:03.077334916 +0000 UTC m=+67.914678173" Jul 14 18:11:03.245942 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.244909 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74756cb6c9-v9rlr"] Jul 14 18:11:03.249919 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.248629 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74756cb6c9-v9rlr" Jul 14 18:11:03.266146 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.253357 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 14 18:11:03.266146 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.253632 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 14 18:11:03.266146 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.253847 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 14 18:11:03.266146 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.254010 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 14 18:11:03.266146 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.254230 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 14 18:11:03.266146 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.254439 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 14 18:11:03.266146 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.254606 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 14 18:11:03.280041 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.279611 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74756cb6c9-v9rlr"] Jul 14 18:11:03.348611 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.348575 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7njb\" (UniqueName: \"kubernetes.io/projected/5e5359d7-520b-40ba-928a-0ec162c0a9bf-kube-api-access-t7njb\") pod \"cluster-proxy-proxy-agent-74756cb6c9-v9rlr\" (UID: \"5e5359d7-520b-40ba-928a-0ec162c0a9bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74756cb6c9-v9rlr" Jul 14 18:11:03.348884 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.348650 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5e5359d7-520b-40ba-928a-0ec162c0a9bf-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-74756cb6c9-v9rlr\" (UID: \"5e5359d7-520b-40ba-928a-0ec162c0a9bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74756cb6c9-v9rlr" Jul 14 18:11:03.348884 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.348702 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e5359d7-520b-40ba-928a-0ec162c0a9bf-additional-service-ca\") pod \"cluster-proxy-proxy-agent-74756cb6c9-v9rlr\" (UID: \"5e5359d7-520b-40ba-928a-0ec162c0a9bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74756cb6c9-v9rlr" Jul 14 18:11:03.348884 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.348743 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5e5359d7-520b-40ba-928a-0ec162c0a9bf-hub\") pod \"cluster-proxy-proxy-agent-74756cb6c9-v9rlr\" (UID: \"5e5359d7-520b-40ba-928a-0ec162c0a9bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74756cb6c9-v9rlr" Jul 14 18:11:03.348884 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.348782 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e5359d7-520b-40ba-928a-0ec162c0a9bf-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-74756cb6c9-v9rlr\" (UID: \"5e5359d7-520b-40ba-928a-0ec162c0a9bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74756cb6c9-v9rlr" Jul 14 18:11:03.348884 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.348818 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5e5359d7-520b-40ba-928a-0ec162c0a9bf-ca\") pod \"cluster-proxy-proxy-agent-74756cb6c9-v9rlr\" (UID: \"5e5359d7-520b-40ba-928a-0ec162c0a9bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74756cb6c9-v9rlr" Jul 14 18:11:03.449547 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.449514 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5e5359d7-520b-40ba-928a-0ec162c0a9bf-ca\") pod \"cluster-proxy-proxy-agent-74756cb6c9-v9rlr\" (UID: \"5e5359d7-520b-40ba-928a-0ec162c0a9bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74756cb6c9-v9rlr" Jul 14 18:11:03.449748 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.449588 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7njb\" (UniqueName: \"kubernetes.io/projected/5e5359d7-520b-40ba-928a-0ec162c0a9bf-kube-api-access-t7njb\") pod \"cluster-proxy-proxy-agent-74756cb6c9-v9rlr\" (UID: \"5e5359d7-520b-40ba-928a-0ec162c0a9bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74756cb6c9-v9rlr" Jul 14 18:11:03.449748 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.449632 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5e5359d7-520b-40ba-928a-0ec162c0a9bf-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-74756cb6c9-v9rlr\" (UID: \"5e5359d7-520b-40ba-928a-0ec162c0a9bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74756cb6c9-v9rlr" Jul 14 18:11:03.449748 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.449668 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e5359d7-520b-40ba-928a-0ec162c0a9bf-additional-service-ca\") pod \"cluster-proxy-proxy-agent-74756cb6c9-v9rlr\" (UID: \"5e5359d7-520b-40ba-928a-0ec162c0a9bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74756cb6c9-v9rlr" Jul 14 18:11:03.449748 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.449711 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5e5359d7-520b-40ba-928a-0ec162c0a9bf-hub\") pod \"cluster-proxy-proxy-agent-74756cb6c9-v9rlr\" (UID: \"5e5359d7-520b-40ba-928a-0ec162c0a9bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74756cb6c9-v9rlr" Jul 14 18:11:03.450117 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.449753 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e5359d7-520b-40ba-928a-0ec162c0a9bf-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-74756cb6c9-v9rlr\" (UID: \"5e5359d7-520b-40ba-928a-0ec162c0a9bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74756cb6c9-v9rlr" Jul 14 18:11:03.451822 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.451221 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e5359d7-520b-40ba-928a-0ec162c0a9bf-additional-service-ca\") pod \"cluster-proxy-proxy-agent-74756cb6c9-v9rlr\" (UID: \"5e5359d7-520b-40ba-928a-0ec162c0a9bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74756cb6c9-v9rlr" Jul 14 18:11:03.456232 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.456187 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e5359d7-520b-40ba-928a-0ec162c0a9bf-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-74756cb6c9-v9rlr\" (UID: \"5e5359d7-520b-40ba-928a-0ec162c0a9bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74756cb6c9-v9rlr" Jul 14 18:11:03.459865 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.459837 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5e5359d7-520b-40ba-928a-0ec162c0a9bf-ca\") pod \"cluster-proxy-proxy-agent-74756cb6c9-v9rlr\" (UID: \"5e5359d7-520b-40ba-928a-0ec162c0a9bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74756cb6c9-v9rlr" Jul 14 18:11:03.461384 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.461355 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5e5359d7-520b-40ba-928a-0ec162c0a9bf-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-74756cb6c9-v9rlr\" (UID: \"5e5359d7-520b-40ba-928a-0ec162c0a9bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74756cb6c9-v9rlr" Jul 14 18:11:03.464115 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.464074 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5e5359d7-520b-40ba-928a-0ec162c0a9bf-hub\") pod \"cluster-proxy-proxy-agent-74756cb6c9-v9rlr\" (UID: \"5e5359d7-520b-40ba-928a-0ec162c0a9bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74756cb6c9-v9rlr" Jul 14 18:11:03.466166 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.466124 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7njb\" (UniqueName: \"kubernetes.io/projected/5e5359d7-520b-40ba-928a-0ec162c0a9bf-kube-api-access-t7njb\") pod \"cluster-proxy-proxy-agent-74756cb6c9-v9rlr\" (UID: \"5e5359d7-520b-40ba-928a-0ec162c0a9bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74756cb6c9-v9rlr" Jul 14 18:11:03.581412 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.581368 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74756cb6c9-v9rlr" Jul 14 18:11:03.781597 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.781357 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:11:03.784479 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:03.784248 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:11:04.028408 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:04.028163 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rbgtx" event={"ID":"a1fc5b88-ec77-4f28-83ff-82725cf2a5e1","Type":"ContainerStarted","Data":"908ecbf0436d55680f18b50bde64f595ad78c09529c4e7d88d688593b640715d"} Jul 14 18:11:04.028408 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:04.028270 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rbgtx" event={"ID":"a1fc5b88-ec77-4f28-83ff-82725cf2a5e1","Type":"ContainerStarted","Data":"6046ac0d91d9d2ba19c2889d836affa1ef31881d6e69913f267756dc01c50ee3"} Jul 14 18:11:04.029375 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:04.029313 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:11:04.030369 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:04.030346 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5977bb4ff6-wbzmj" Jul 14 18:11:04.056757 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:04.055875 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rbgtx" podStartSLOduration=66.751169402 podStartE2EDuration="1m8.05585805s" totalImagesPullingTime="1.304688648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:09:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:04.055181626 +0000 UTC m=+68.892524885" watchObservedRunningTime="2026-07-14 18:11:04.05585805 +0000 UTC m=+68.893201397" Jul 14 18:11:05.755925 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:05.755551 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74756cb6c9-v9rlr"] Jul 14 18:11:06.038003 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:06.037966 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7jp42" event={"ID":"1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452","Type":"ContainerStarted","Data":"f5c9529be6e5892b27518371fca2ec2be8cab1b53842048516e6806868361762"} Jul 14 18:11:06.039835 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:06.039785 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-lgbnj" event={"ID":"3dd54969-694e-4b8c-8a8c-21412a2c954e","Type":"ContainerStarted","Data":"df8a97f959a87168e174b9e498b5f96a03ba4b5e05ff8a65dcf60d6b1d054c83"} Jul 14 18:11:06.041779 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:06.041757 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-n4l4d" event={"ID":"5d052fcc-79b5-45cd-a58e-6bb582e25ab8","Type":"ContainerStarted","Data":"b36e1c4b64e868021e89ba3b1ea4e29a35f3106ded820fb963bda9e2a6e99951"} Jul 14 18:11:06.041914 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:06.041841 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-n4l4d" event={"ID":"5d052fcc-79b5-45cd-a58e-6bb582e25ab8","Type":"ContainerStarted","Data":"8f587f9be9e5012234481122dbddb9c2a19c61656f270e55d2b47cfc261cea0f"} Jul 14 18:11:06.044601 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:06.044472 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6k9qk" event={"ID":"44267c25-6aee-4e48-93da-3b222a618eff","Type":"ContainerStarted","Data":"592900daa25f82126c8197656a828ebd4d8d1d15a2605fce0cfedf40eabc2c17"} Jul 14 18:11:06.044601 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:06.044517 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6k9qk" event={"ID":"44267c25-6aee-4e48-93da-3b222a618eff","Type":"ContainerStarted","Data":"11fe5c1483ad8fa01ec0f06fcd1328057c937f65c9a637b7cdab20db08bb67f2"} Jul 14 18:11:06.044601 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:06.044599 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6k9qk" Jul 14 18:11:06.050202 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:06.046363 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74756cb6c9-v9rlr" event={"ID":"5e5359d7-520b-40ba-928a-0ec162c0a9bf","Type":"ContainerStarted","Data":"62cc918950419950d7a6fa118dd779942c20bd13bbb6d5ce5d4ff49e21b7a2d3"} Jul 14 18:11:06.063621 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:06.063562 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-7jp42" podStartSLOduration=58.372268678 podStartE2EDuration="1m2.063545674s" totalImagesPullingTime="3.691276996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:10:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:06.062745049 +0000 UTC m=+70.900088332" watchObservedRunningTime="2026-07-14 18:11:06.063545674 +0000 UTC m=+70.900888933" Jul 14 18:11:06.082580 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:06.082525 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-lgbnj" podStartSLOduration=51.578815424 podStartE2EDuration="55.08251181s" totalImagesPullingTime="3.503696386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:10:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:06.082486331 +0000 UTC m=+70.919829588" watchObservedRunningTime="2026-07-14 18:11:06.08251181 +0000 UTC m=+70.919855067" Jul 14 18:11:06.117831 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:06.117738 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6k9qk" podStartSLOduration=33.504339687 podStartE2EDuration="37.117722489s" totalImagesPullingTime="3.613382802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:10:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:06.117093092 +0000 UTC m=+70.954436350" watchObservedRunningTime="2026-07-14 18:11:06.117722489 +0000 UTC m=+70.955065812" Jul 14 18:11:06.144636 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:06.144569 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-n4l4d" podStartSLOduration=58.564137912 podStartE2EDuration="1m2.144547976s" totalImagesPullingTime="3.580410064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:10:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:06.143474244 +0000 UTC m=+70.980817503" watchObservedRunningTime="2026-07-14 18:11:06.144547976 +0000 UTC m=+70.981891230" Jul 14 18:11:09.059980 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:09.059930 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6czgz" event={"ID":"015d11c8-2009-4c76-815a-c2d87c42e7b5","Type":"ContainerStarted","Data":"1e8c7edf5c39985290b02767f626c29129542b853678a9dfab0135b2b287431f"} Jul 14 18:11:09.083402 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:09.083346 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6czgz" podStartSLOduration=66.807886125 podStartE2EDuration="1m13.083328009s" totalImagesPullingTime="6.275441884s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:09:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:09.083124772 +0000 UTC m=+73.920468061" watchObservedRunningTime="2026-07-14 18:11:09.083328009 +0000 UTC m=+73.920671266" Jul 14 18:11:11.529993 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:11.529967 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c5e5426-5ad9-4ecc-93aa-577db1551181-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9g9f7\" (UID: \"9c5e5426-5ad9-4ecc-93aa-577db1551181\") " pod="openshift-insights/insights-runtime-extractor-9g9f7" Jul 14 18:11:11.532410 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:11.532386 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c5e5426-5ad9-4ecc-93aa-577db1551181-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9g9f7\" (UID: \"9c5e5426-5ad9-4ecc-93aa-577db1551181\") " pod="openshift-insights/insights-runtime-extractor-9g9f7" Jul 14 18:11:11.774231 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:11.774204 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-266xb\"" Jul 14 18:11:11.781686 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:11.781652 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9g9f7" Jul 14 18:11:11.938660 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:11.938631 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9g9f7"] Jul 14 18:11:11.941401 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:11:11.941380 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c5e5426_5ad9_4ecc_93aa_577db1551181.slice/crio-4dfa8858130533aede47df5011acf28035ae28ffbaefda77ad3e9392237e3174 WatchSource:0}: Error finding container 4dfa8858130533aede47df5011acf28035ae28ffbaefda77ad3e9392237e3174: Status 404 returned error can't find the container with id 4dfa8858130533aede47df5011acf28035ae28ffbaefda77ad3e9392237e3174 Jul 14 18:11:12.072500 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:12.072402 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74756cb6c9-v9rlr" event={"ID":"5e5359d7-520b-40ba-928a-0ec162c0a9bf","Type":"ContainerStarted","Data":"d9c0de0294cf551423f42253dcb36ef822228ce63b689a9d616f9f1480be8e4a"} Jul 14 18:11:12.072500 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:12.072449 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74756cb6c9-v9rlr" event={"ID":"5e5359d7-520b-40ba-928a-0ec162c0a9bf","Type":"ContainerStarted","Data":"2aa4e3b757d1c83a505bd67078ea0af6d7e797e610fb564f51509863e03aae9c"} Jul 14 18:11:12.072500 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:12.072467 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74756cb6c9-v9rlr" event={"ID":"5e5359d7-520b-40ba-928a-0ec162c0a9bf","Type":"ContainerStarted","Data":"bb37fae901f011f0b87a7bc495dec7098a06cade0e9a6e687ed2cb9b4bac90e5"} Jul 14 18:11:12.073516 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:12.073497 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9g9f7" event={"ID":"9c5e5426-5ad9-4ecc-93aa-577db1551181","Type":"ContainerStarted","Data":"4dfa8858130533aede47df5011acf28035ae28ffbaefda77ad3e9392237e3174"} Jul 14 18:11:12.097954 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:12.097902 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74756cb6c9-v9rlr" podStartSLOduration=3.68288501 podStartE2EDuration="9.097886342s" totalImagesPullingTime="5.415001332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:11:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:12.095687215 +0000 UTC m=+76.933030462" watchObservedRunningTime="2026-07-14 18:11:12.097886342 +0000 UTC m=+76.935229600" Jul 14 18:11:16.053515 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:16.052815 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6k9qk" Jul 14 18:11:17.096750 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:17.096716 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9g9f7" event={"ID":"9c5e5426-5ad9-4ecc-93aa-577db1551181","Type":"ContainerStarted","Data":"f7d76c31ea9d7b53ed9f8493895fd631e44ff1640cfcbc044069869ddbd0fae4"} Jul 14 18:11:18.101932 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:18.101894 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9g9f7" event={"ID":"9c5e5426-5ad9-4ecc-93aa-577db1551181","Type":"ContainerStarted","Data":"249f1065003325309f490d15868634ebda4633a626e3fddd7265915ab5fc1e5d"} Jul 14 18:11:21.115657 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:21.115563 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9g9f7" event={"ID":"9c5e5426-5ad9-4ecc-93aa-577db1551181","Type":"ContainerStarted","Data":"ee07ea51ca2127e6c69715772c50ed705a2d49247eb3a6687c5ade13b17a1c1d"} Jul 14 18:11:21.140042 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:21.139992 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9g9f7" podStartSLOduration=33.426966033 podStartE2EDuration="42.139975773s" totalImagesPullingTime="8.71300974s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:10:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:21.139170513 +0000 UTC m=+85.976513792" watchObservedRunningTime="2026-07-14 18:11:21.139975773 +0000 UTC m=+85.977319046" Jul 14 18:11:25.036944 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:25.036915 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:11:25.585608 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:25.585579 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f46bf986b-hh2fk"] Jul 14 18:11:25.873827 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:25.873738 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-phnhn"] Jul 14 18:11:25.879869 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:25.879837 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-phnhn" Jul 14 18:11:25.883330 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:25.883308 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 14 18:11:25.883728 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:25.883707 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pj49j\"" Jul 14 18:11:25.883838 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:25.883818 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 14 18:11:25.884572 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:25.884560 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 14 18:11:25.884619 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:25.884594 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 14 18:11:25.948950 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:25.948911 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09b26111-d3a6-47f0-98eb-73a1f0008b00-metrics-client-ca\") pod \"node-exporter-phnhn\" (UID: \"09b26111-d3a6-47f0-98eb-73a1f0008b00\") " pod="openshift-monitoring/node-exporter-phnhn" Jul 14 18:11:25.949138 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:25.948962 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09b26111-d3a6-47f0-98eb-73a1f0008b00-sys\") pod \"node-exporter-phnhn\" (UID: \"09b26111-d3a6-47f0-98eb-73a1f0008b00\") " pod="openshift-monitoring/node-exporter-phnhn" Jul 14 18:11:25.949138 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:25.948985 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09b26111-d3a6-47f0-98eb-73a1f0008b00-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-phnhn\" (UID: \"09b26111-d3a6-47f0-98eb-73a1f0008b00\") " pod="openshift-monitoring/node-exporter-phnhn" Jul 14 18:11:25.949138 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:25.949072 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9pnq\" (UniqueName: \"kubernetes.io/projected/09b26111-d3a6-47f0-98eb-73a1f0008b00-kube-api-access-n9pnq\") pod \"node-exporter-phnhn\" (UID: \"09b26111-d3a6-47f0-98eb-73a1f0008b00\") " pod="openshift-monitoring/node-exporter-phnhn" Jul 14 18:11:25.949138 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:25.949129 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/09b26111-d3a6-47f0-98eb-73a1f0008b00-node-exporter-textfile\") pod \"node-exporter-phnhn\" (UID: \"09b26111-d3a6-47f0-98eb-73a1f0008b00\") " pod="openshift-monitoring/node-exporter-phnhn" Jul 14 18:11:25.949317 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:25.949156 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/09b26111-d3a6-47f0-98eb-73a1f0008b00-node-exporter-accelerators-collector-config\") pod \"node-exporter-phnhn\" (UID: \"09b26111-d3a6-47f0-98eb-73a1f0008b00\") " pod="openshift-monitoring/node-exporter-phnhn" Jul 14 18:11:25.949317 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:25.949176 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/09b26111-d3a6-47f0-98eb-73a1f0008b00-root\") pod \"node-exporter-phnhn\" (UID: \"09b26111-d3a6-47f0-98eb-73a1f0008b00\") " pod="openshift-monitoring/node-exporter-phnhn" Jul 14 18:11:25.949317 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:25.949213 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/09b26111-d3a6-47f0-98eb-73a1f0008b00-node-exporter-tls\") pod \"node-exporter-phnhn\" (UID: \"09b26111-d3a6-47f0-98eb-73a1f0008b00\") " pod="openshift-monitoring/node-exporter-phnhn" Jul 14 18:11:25.949317 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:25.949285 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/09b26111-d3a6-47f0-98eb-73a1f0008b00-node-exporter-wtmp\") pod \"node-exporter-phnhn\" (UID: \"09b26111-d3a6-47f0-98eb-73a1f0008b00\") " pod="openshift-monitoring/node-exporter-phnhn" Jul 14 18:11:26.050361 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:26.050331 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09b26111-d3a6-47f0-98eb-73a1f0008b00-sys\") pod \"node-exporter-phnhn\" (UID: \"09b26111-d3a6-47f0-98eb-73a1f0008b00\") " pod="openshift-monitoring/node-exporter-phnhn" Jul 14 18:11:26.050361 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:26.050361 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09b26111-d3a6-47f0-98eb-73a1f0008b00-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-phnhn\" (UID: \"09b26111-d3a6-47f0-98eb-73a1f0008b00\") " pod="openshift-monitoring/node-exporter-phnhn" Jul 14 18:11:26.050837 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:26.050383 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9pnq\" (UniqueName: \"kubernetes.io/projected/09b26111-d3a6-47f0-98eb-73a1f0008b00-kube-api-access-n9pnq\") pod \"node-exporter-phnhn\" (UID: \"09b26111-d3a6-47f0-98eb-73a1f0008b00\") " pod="openshift-monitoring/node-exporter-phnhn" Jul 14 18:11:26.050837 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:26.050414 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/09b26111-d3a6-47f0-98eb-73a1f0008b00-node-exporter-textfile\") pod \"node-exporter-phnhn\" (UID: \"09b26111-d3a6-47f0-98eb-73a1f0008b00\") " pod="openshift-monitoring/node-exporter-phnhn" Jul 14 18:11:26.050837 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:26.050442 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/09b26111-d3a6-47f0-98eb-73a1f0008b00-node-exporter-accelerators-collector-config\") pod \"node-exporter-phnhn\" (UID: \"09b26111-d3a6-47f0-98eb-73a1f0008b00\") " pod="openshift-monitoring/node-exporter-phnhn" Jul 14 18:11:26.050837 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:26.050452 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09b26111-d3a6-47f0-98eb-73a1f0008b00-sys\") pod \"node-exporter-phnhn\" (UID: \"09b26111-d3a6-47f0-98eb-73a1f0008b00\") " pod="openshift-monitoring/node-exporter-phnhn" Jul 14 18:11:26.050837 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:26.050465 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/09b26111-d3a6-47f0-98eb-73a1f0008b00-root\") pod \"node-exporter-phnhn\" (UID: \"09b26111-d3a6-47f0-98eb-73a1f0008b00\") " pod="openshift-monitoring/node-exporter-phnhn" Jul 14 18:11:26.050837 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:26.050504 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/09b26111-d3a6-47f0-98eb-73a1f0008b00-root\") pod \"node-exporter-phnhn\" (UID: \"09b26111-d3a6-47f0-98eb-73a1f0008b00\") " pod="openshift-monitoring/node-exporter-phnhn" Jul 14 18:11:26.050837 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:26.050553 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/09b26111-d3a6-47f0-98eb-73a1f0008b00-node-exporter-tls\") pod \"node-exporter-phnhn\" (UID: \"09b26111-d3a6-47f0-98eb-73a1f0008b00\") " pod="openshift-monitoring/node-exporter-phnhn" Jul 14 18:11:26.050837 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:26.050602 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/09b26111-d3a6-47f0-98eb-73a1f0008b00-node-exporter-wtmp\") pod \"node-exporter-phnhn\" (UID: \"09b26111-d3a6-47f0-98eb-73a1f0008b00\") " pod="openshift-monitoring/node-exporter-phnhn" Jul 14 18:11:26.050837 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:26.050636 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09b26111-d3a6-47f0-98eb-73a1f0008b00-metrics-client-ca\") pod \"node-exporter-phnhn\" (UID: \"09b26111-d3a6-47f0-98eb-73a1f0008b00\") " pod="openshift-monitoring/node-exporter-phnhn" Jul 14 18:11:26.050837 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:11:26.050671 2565 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 14 18:11:26.050837 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:11:26.050743 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09b26111-d3a6-47f0-98eb-73a1f0008b00-node-exporter-tls podName:09b26111-d3a6-47f0-98eb-73a1f0008b00 nodeName:}" failed. No retries permitted until 2026-07-14 18:11:26.55071992 +0000 UTC m=+91.388063156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/09b26111-d3a6-47f0-98eb-73a1f0008b00-node-exporter-tls") pod "node-exporter-phnhn" (UID: "09b26111-d3a6-47f0-98eb-73a1f0008b00") : secret "node-exporter-tls" not found Jul 14 18:11:26.050837 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:26.050779 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/09b26111-d3a6-47f0-98eb-73a1f0008b00-node-exporter-textfile\") pod \"node-exporter-phnhn\" (UID: \"09b26111-d3a6-47f0-98eb-73a1f0008b00\") " pod="openshift-monitoring/node-exporter-phnhn" Jul 14 18:11:26.050837 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:26.050801 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/09b26111-d3a6-47f0-98eb-73a1f0008b00-node-exporter-wtmp\") pod \"node-exporter-phnhn\" (UID: \"09b26111-d3a6-47f0-98eb-73a1f0008b00\") " pod="openshift-monitoring/node-exporter-phnhn" Jul 14 18:11:26.051405 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:26.051184 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09b26111-d3a6-47f0-98eb-73a1f0008b00-metrics-client-ca\") pod \"node-exporter-phnhn\" (UID: \"09b26111-d3a6-47f0-98eb-73a1f0008b00\") " pod="openshift-monitoring/node-exporter-phnhn" Jul 14 18:11:26.051405 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:26.051183 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/09b26111-d3a6-47f0-98eb-73a1f0008b00-node-exporter-accelerators-collector-config\") pod \"node-exporter-phnhn\" (UID: \"09b26111-d3a6-47f0-98eb-73a1f0008b00\") " pod="openshift-monitoring/node-exporter-phnhn" Jul 14 18:11:26.052831 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:26.052812 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09b26111-d3a6-47f0-98eb-73a1f0008b00-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-phnhn\" (UID: \"09b26111-d3a6-47f0-98eb-73a1f0008b00\") " pod="openshift-monitoring/node-exporter-phnhn" Jul 14 18:11:26.060005 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:26.059977 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9pnq\" (UniqueName: \"kubernetes.io/projected/09b26111-d3a6-47f0-98eb-73a1f0008b00-kube-api-access-n9pnq\") pod \"node-exporter-phnhn\" (UID: \"09b26111-d3a6-47f0-98eb-73a1f0008b00\") " pod="openshift-monitoring/node-exporter-phnhn" Jul 14 18:11:26.556156 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:26.556114 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/09b26111-d3a6-47f0-98eb-73a1f0008b00-node-exporter-tls\") pod \"node-exporter-phnhn\" (UID: \"09b26111-d3a6-47f0-98eb-73a1f0008b00\") " pod="openshift-monitoring/node-exporter-phnhn" Jul 14 18:11:26.558517 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:26.558490 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/09b26111-d3a6-47f0-98eb-73a1f0008b00-node-exporter-tls\") pod \"node-exporter-phnhn\" (UID: \"09b26111-d3a6-47f0-98eb-73a1f0008b00\") " pod="openshift-monitoring/node-exporter-phnhn" Jul 14 18:11:26.788816 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:26.788774 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-phnhn" Jul 14 18:11:26.802811 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:11:26.802774 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b26111_d3a6_47f0_98eb_73a1f0008b00.slice/crio-c5f5ae7f5ef0419a76d8ac9b5274a786cfda7f19ceed5949db232c62150a7a13 WatchSource:0}: Error finding container c5f5ae7f5ef0419a76d8ac9b5274a786cfda7f19ceed5949db232c62150a7a13: Status 404 returned error can't find the container with id c5f5ae7f5ef0419a76d8ac9b5274a786cfda7f19ceed5949db232c62150a7a13 Jul 14 18:11:27.138542 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:27.138445 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-phnhn" event={"ID":"09b26111-d3a6-47f0-98eb-73a1f0008b00","Type":"ContainerStarted","Data":"c5f5ae7f5ef0419a76d8ac9b5274a786cfda7f19ceed5949db232c62150a7a13"} Jul 14 18:11:28.143863 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:28.143750 2565 generic.go:356] "Generic (PLEG): container finished" podID="09b26111-d3a6-47f0-98eb-73a1f0008b00" containerID="db07ef8fbc12bc70ef3b8eba4a6233fa200bdd1f20755d4854cbc333e76f8c1d" exitCode=0 Jul 14 18:11:28.143863 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:28.143821 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-phnhn" event={"ID":"09b26111-d3a6-47f0-98eb-73a1f0008b00","Type":"ContainerDied","Data":"db07ef8fbc12bc70ef3b8eba4a6233fa200bdd1f20755d4854cbc333e76f8c1d"} Jul 14 18:11:29.150718 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:29.150679 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-phnhn" event={"ID":"09b26111-d3a6-47f0-98eb-73a1f0008b00","Type":"ContainerStarted","Data":"99f628396b22827264c91fd7f6d81b8a68889d1ab2cf2faff089603a8241b888"} Jul 14 18:11:29.151194 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:29.150730 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-phnhn" event={"ID":"09b26111-d3a6-47f0-98eb-73a1f0008b00","Type":"ContainerStarted","Data":"ddcaac1ded48e93cab70b82b0c4c0544e7130c05f8bc824717c45deed604d533"} Jul 14 18:11:29.176117 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:29.176061 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-phnhn" podStartSLOduration=3.022446785 podStartE2EDuration="4.176043389s" totalImagesPullingTime="1.079665359s" totalInitContainerRuntime="73.931245ms" isStatefulPod=true podCreationTimestamp="2026-07-14 18:11:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:29.173924752 +0000 UTC m=+94.011268028" watchObservedRunningTime="2026-07-14 18:11:29.176043389 +0000 UTC m=+94.013386684" Jul 14 18:11:34.031579 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:34.031549 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-x92qm" Jul 14 18:11:44.209612 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:44.209578 2565 generic.go:356] "Generic (PLEG): container finished" podID="29cfec31-7adb-41a3-965d-11c1d199b9c3" containerID="175debad6ded35d41fe5ff7266f7c9c82917cd7f86cb713e2e69b2e7ff00a122" exitCode=0 Jul 14 18:11:44.210006 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:44.209637 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-mqqk9" event={"ID":"29cfec31-7adb-41a3-965d-11c1d199b9c3","Type":"ContainerDied","Data":"175debad6ded35d41fe5ff7266f7c9c82917cd7f86cb713e2e69b2e7ff00a122"} Jul 14 18:11:44.210006 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:44.209964 2565 scope.go:117] "RemoveContainer" containerID="175debad6ded35d41fe5ff7266f7c9c82917cd7f86cb713e2e69b2e7ff00a122" Jul 14 18:11:44.908470 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:44.908446 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5977bb4ff6-wbzmj_7b2746f9-dc43-4311-aa5b-ad3bc351deb3/router/0.log" Jul 14 18:11:44.927479 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:44.927451 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jmt26_211b531e-f105-44a4-934f-d2e97e981b84/serve-healthcheck-canary/0.log" Jul 14 18:11:45.214883 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:45.214784 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-mqqk9" event={"ID":"29cfec31-7adb-41a3-965d-11c1d199b9c3","Type":"ContainerStarted","Data":"e1e305df4b6a2cdc13cc2e8f0b6d280fc46b357991ba9d9cb4367155073881d4"} Jul 14 18:11:49.231764 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:49.231727 2565 generic.go:356] "Generic (PLEG): container finished" podID="a2dc349f-af57-4915-b047-20b77acd7ac9" containerID="4362c8e05ab6a4a1306e86cbd626b2def7279fa5548cf040ca0c0414ed5cebdb" exitCode=0 Jul 14 18:11:49.232158 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:49.231783 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zd4lp" event={"ID":"a2dc349f-af57-4915-b047-20b77acd7ac9","Type":"ContainerDied","Data":"4362c8e05ab6a4a1306e86cbd626b2def7279fa5548cf040ca0c0414ed5cebdb"} Jul 14 18:11:49.232158 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:49.232092 2565 scope.go:117] "RemoveContainer" containerID="4362c8e05ab6a4a1306e86cbd626b2def7279fa5548cf040ca0c0414ed5cebdb" Jul 14 18:11:50.237066 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:50.237026 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zd4lp" event={"ID":"a2dc349f-af57-4915-b047-20b77acd7ac9","Type":"ContainerStarted","Data":"91b865a200285dba98854274e4059fac3992c37a28a1da2e09c0bbff2fbccc7c"} Jul 14 18:11:50.606054 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:50.606014 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" podUID="bd03d2f0-2403-414f-b02e-ec3c5e7fb214" containerName="registry" containerID="cri-o://92d2306188f1833d505a7cda6506114a9fe16699fed5b21be9e179ba1e3244bb" gracePeriod=30 Jul 14 18:11:50.866524 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:50.866467 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:11:50.949746 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:50.949719 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-image-registry-private-configuration\") pod \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " Jul 14 18:11:50.949887 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:50.949753 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-bound-sa-token\") pod \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " Jul 14 18:11:50.949887 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:50.949778 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-ca-trust-extracted\") pod \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " Jul 14 18:11:50.949887 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:50.949798 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-registry-tls\") pod \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " Jul 14 18:11:50.949887 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:50.949833 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dljfz\" (UniqueName: \"kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-kube-api-access-dljfz\") pod \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " Jul 14 18:11:50.949887 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:50.949858 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-registry-certificates\") pod \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " Jul 14 18:11:50.949887 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:50.949885 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-installation-pull-secrets\") pod \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " Jul 14 18:11:50.950219 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:50.949903 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-trusted-ca\") pod \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\" (UID: \"bd03d2f0-2403-414f-b02e-ec3c5e7fb214\") " Jul 14 18:11:50.950767 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:50.950672 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "bd03d2f0-2403-414f-b02e-ec3c5e7fb214" (UID: "bd03d2f0-2403-414f-b02e-ec3c5e7fb214"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:11:50.950767 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:50.950737 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "bd03d2f0-2403-414f-b02e-ec3c5e7fb214" (UID: "bd03d2f0-2403-414f-b02e-ec3c5e7fb214"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:11:50.952729 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:50.952694 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-kube-api-access-dljfz" (OuterVolumeSpecName: "kube-api-access-dljfz") pod "bd03d2f0-2403-414f-b02e-ec3c5e7fb214" (UID: "bd03d2f0-2403-414f-b02e-ec3c5e7fb214"). InnerVolumeSpecName "kube-api-access-dljfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:11:50.952846 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:50.952809 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "bd03d2f0-2403-414f-b02e-ec3c5e7fb214" (UID: "bd03d2f0-2403-414f-b02e-ec3c5e7fb214"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:11:50.952923 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:50.952823 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "bd03d2f0-2403-414f-b02e-ec3c5e7fb214" (UID: "bd03d2f0-2403-414f-b02e-ec3c5e7fb214"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:11:50.952923 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:50.952912 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bd03d2f0-2403-414f-b02e-ec3c5e7fb214" (UID: "bd03d2f0-2403-414f-b02e-ec3c5e7fb214"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:11:50.953015 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:50.952943 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "bd03d2f0-2403-414f-b02e-ec3c5e7fb214" (UID: "bd03d2f0-2403-414f-b02e-ec3c5e7fb214"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:11:50.958604 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:50.958580 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "bd03d2f0-2403-414f-b02e-ec3c5e7fb214" (UID: "bd03d2f0-2403-414f-b02e-ec3c5e7fb214"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:11:51.051118 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:51.051070 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dljfz\" (UniqueName: \"kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-kube-api-access-dljfz\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:11:51.051118 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:51.051117 2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-registry-certificates\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:11:51.051290 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:51.051132 2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-installation-pull-secrets\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:11:51.051290 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:51.051145 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-trusted-ca\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:11:51.051290 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:51.051159 2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-image-registry-private-configuration\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:11:51.051290 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:51.051169 2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-bound-sa-token\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:11:51.051290 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:51.051177 2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-ca-trust-extracted\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:11:51.051290 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:51.051186 2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd03d2f0-2403-414f-b02e-ec3c5e7fb214-registry-tls\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:11:51.241665 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:51.241575 2565 generic.go:356] "Generic (PLEG): container finished" podID="bd03d2f0-2403-414f-b02e-ec3c5e7fb214" containerID="92d2306188f1833d505a7cda6506114a9fe16699fed5b21be9e179ba1e3244bb" exitCode=0 Jul 14 18:11:51.241665 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:51.241621 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" Jul 14 18:11:51.242137 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:51.241660 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" event={"ID":"bd03d2f0-2403-414f-b02e-ec3c5e7fb214","Type":"ContainerDied","Data":"92d2306188f1833d505a7cda6506114a9fe16699fed5b21be9e179ba1e3244bb"} Jul 14 18:11:51.242137 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:51.241716 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f46bf986b-hh2fk" event={"ID":"bd03d2f0-2403-414f-b02e-ec3c5e7fb214","Type":"ContainerDied","Data":"f0258f2cd3206ff0ce1fbae5fe12f7b6a8d351885dabdb76293dcb1bf1dad4c0"} Jul 14 18:11:51.242137 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:51.241743 2565 scope.go:117] "RemoveContainer" containerID="92d2306188f1833d505a7cda6506114a9fe16699fed5b21be9e179ba1e3244bb" Jul 14 18:11:51.250348 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:51.250332 2565 scope.go:117] "RemoveContainer" containerID="92d2306188f1833d505a7cda6506114a9fe16699fed5b21be9e179ba1e3244bb" Jul 14 18:11:51.250614 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:11:51.250594 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92d2306188f1833d505a7cda6506114a9fe16699fed5b21be9e179ba1e3244bb\": container with ID starting with 92d2306188f1833d505a7cda6506114a9fe16699fed5b21be9e179ba1e3244bb not found: ID does not exist" containerID="92d2306188f1833d505a7cda6506114a9fe16699fed5b21be9e179ba1e3244bb" Jul 14 18:11:51.250691 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:51.250627 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92d2306188f1833d505a7cda6506114a9fe16699fed5b21be9e179ba1e3244bb"} err="failed to get container status \"92d2306188f1833d505a7cda6506114a9fe16699fed5b21be9e179ba1e3244bb\": rpc error: code = NotFound desc = could not find container \"92d2306188f1833d505a7cda6506114a9fe16699fed5b21be9e179ba1e3244bb\": container with ID starting with 92d2306188f1833d505a7cda6506114a9fe16699fed5b21be9e179ba1e3244bb not found: ID does not exist" Jul 14 18:11:51.264574 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:51.264549 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f46bf986b-hh2fk"] Jul 14 18:11:51.271296 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:51.271274 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5f46bf986b-hh2fk"] Jul 14 18:11:51.688614 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:11:51.688585 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd03d2f0-2403-414f-b02e-ec3c5e7fb214" path="/var/lib/kubelet/pods/bd03d2f0-2403-414f-b02e-ec3c5e7fb214/volumes" Jul 14 18:12:04.295253 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:12:04.295215 2565 generic.go:356] "Generic (PLEG): container finished" podID="3f335f2a-958b-49ca-9c11-db21d1694db3" containerID="6fcf88800a523eca2a11e8010d2997175bc6f437b36fd7fdfaa40e47f908a887" exitCode=0 Jul 14 18:12:04.295658 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:12:04.295283 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-7kfjz" event={"ID":"3f335f2a-958b-49ca-9c11-db21d1694db3","Type":"ContainerDied","Data":"6fcf88800a523eca2a11e8010d2997175bc6f437b36fd7fdfaa40e47f908a887"} Jul 14 18:12:04.295658 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:12:04.295589 2565 scope.go:117] "RemoveContainer" containerID="6fcf88800a523eca2a11e8010d2997175bc6f437b36fd7fdfaa40e47f908a887" Jul 14 18:12:05.301640 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:12:05.301612 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-7kfjz" event={"ID":"3f335f2a-958b-49ca-9c11-db21d1694db3","Type":"ContainerStarted","Data":"05684532ce2388e85d1768b930da2617ece8edf6f28b24385239d6c7c31882ba"} Jul 14 18:14:55.577373 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:14:55.577331 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wqpt2_4525dcda-ead6-4800-8d17-58812c289010/console-operator/1.log" Jul 14 18:14:55.580575 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:14:55.580551 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f99bz_5f0ecb09-5c7f-4709-9b4a-1a6a03580822/ovn-acl-logging/0.log" Jul 14 18:14:55.582502 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:14:55.582484 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wqpt2_4525dcda-ead6-4800-8d17-58812c289010/console-operator/1.log" Jul 14 18:14:55.585569 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:14:55.585553 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f99bz_5f0ecb09-5c7f-4709-9b4a-1a6a03580822/ovn-acl-logging/0.log" Jul 14 18:14:55.587229 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:14:55.587210 2565 kubelet.go:1648] "Image garbage collection succeeded" Jul 14 18:15:05.114463 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:05.114425 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-557b7f646c-84xtc"] Jul 14 18:15:05.116826 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:05.114880 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd03d2f0-2403-414f-b02e-ec3c5e7fb214" containerName="registry" Jul 14 18:15:05.116826 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:05.114899 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd03d2f0-2403-414f-b02e-ec3c5e7fb214" containerName="registry" Jul 14 18:15:05.116826 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:05.114949 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd03d2f0-2403-414f-b02e-ec3c5e7fb214" containerName="registry" Jul 14 18:15:05.117687 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:05.117669 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-84xtc" Jul 14 18:15:05.121964 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:05.121944 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 14 18:15:05.122048 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:05.122025 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-52hn2\"" Jul 14 18:15:05.122671 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:05.122637 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 14 18:15:05.122990 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:05.122974 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 14 18:15:05.123076 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:05.123053 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 14 18:15:05.191007 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:05.190978 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a2cc16bb-ccb9-4852-aaa2-00a337a95ee2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-557b7f646c-84xtc\" (UID: \"a2cc16bb-ccb9-4852-aaa2-00a337a95ee2\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-84xtc" Jul 14 18:15:05.191147 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:05.191010 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a2cc16bb-ccb9-4852-aaa2-00a337a95ee2-manifests\") pod \"opendatahub-operator-controller-manager-557b7f646c-84xtc\" (UID: \"a2cc16bb-ccb9-4852-aaa2-00a337a95ee2\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-84xtc" Jul 14 18:15:05.191147 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:05.191053 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4d4j\" (UniqueName: \"kubernetes.io/projected/a2cc16bb-ccb9-4852-aaa2-00a337a95ee2-kube-api-access-z4d4j\") pod \"opendatahub-operator-controller-manager-557b7f646c-84xtc\" (UID: \"a2cc16bb-ccb9-4852-aaa2-00a337a95ee2\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-84xtc" Jul 14 18:15:05.191147 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:05.191128 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2cc16bb-ccb9-4852-aaa2-00a337a95ee2-webhook-cert\") pod \"opendatahub-operator-controller-manager-557b7f646c-84xtc\" (UID: \"a2cc16bb-ccb9-4852-aaa2-00a337a95ee2\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-84xtc" Jul 14 18:15:05.191270 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:05.191150 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2cc16bb-ccb9-4852-aaa2-00a337a95ee2-tmp\") pod \"opendatahub-operator-controller-manager-557b7f646c-84xtc\" (UID: \"a2cc16bb-ccb9-4852-aaa2-00a337a95ee2\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-84xtc" Jul 14 18:15:05.218489 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:05.218468 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-557b7f646c-84xtc"] Jul 14 18:15:05.292211 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:05.292179 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4d4j\" (UniqueName: \"kubernetes.io/projected/a2cc16bb-ccb9-4852-aaa2-00a337a95ee2-kube-api-access-z4d4j\") pod \"opendatahub-operator-controller-manager-557b7f646c-84xtc\" (UID: \"a2cc16bb-ccb9-4852-aaa2-00a337a95ee2\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-84xtc" Jul 14 18:15:05.292367 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:05.292236 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2cc16bb-ccb9-4852-aaa2-00a337a95ee2-webhook-cert\") pod \"opendatahub-operator-controller-manager-557b7f646c-84xtc\" (UID: \"a2cc16bb-ccb9-4852-aaa2-00a337a95ee2\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-84xtc" Jul 14 18:15:05.292367 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:05.292262 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2cc16bb-ccb9-4852-aaa2-00a337a95ee2-tmp\") pod \"opendatahub-operator-controller-manager-557b7f646c-84xtc\" (UID: \"a2cc16bb-ccb9-4852-aaa2-00a337a95ee2\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-84xtc" Jul 14 18:15:05.292367 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:05.292299 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a2cc16bb-ccb9-4852-aaa2-00a337a95ee2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-557b7f646c-84xtc\" (UID: \"a2cc16bb-ccb9-4852-aaa2-00a337a95ee2\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-84xtc" Jul 14 18:15:05.292367 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:05.292323 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a2cc16bb-ccb9-4852-aaa2-00a337a95ee2-manifests\") pod \"opendatahub-operator-controller-manager-557b7f646c-84xtc\" (UID: \"a2cc16bb-ccb9-4852-aaa2-00a337a95ee2\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-84xtc" Jul 14 18:15:05.292723 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:05.292701 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a2cc16bb-ccb9-4852-aaa2-00a337a95ee2-manifests\") pod \"opendatahub-operator-controller-manager-557b7f646c-84xtc\" (UID: \"a2cc16bb-ccb9-4852-aaa2-00a337a95ee2\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-84xtc" Jul 14 18:15:05.294585 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:05.294565 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2cc16bb-ccb9-4852-aaa2-00a337a95ee2-tmp\") pod \"opendatahub-operator-controller-manager-557b7f646c-84xtc\" (UID: \"a2cc16bb-ccb9-4852-aaa2-00a337a95ee2\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-84xtc" Jul 14 18:15:05.294834 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:05.294816 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2cc16bb-ccb9-4852-aaa2-00a337a95ee2-webhook-cert\") pod \"opendatahub-operator-controller-manager-557b7f646c-84xtc\" (UID: \"a2cc16bb-ccb9-4852-aaa2-00a337a95ee2\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-84xtc" Jul 14 18:15:05.294921 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:05.294900 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a2cc16bb-ccb9-4852-aaa2-00a337a95ee2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-557b7f646c-84xtc\" (UID: \"a2cc16bb-ccb9-4852-aaa2-00a337a95ee2\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-84xtc" Jul 14 18:15:05.304308 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:05.304287 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4d4j\" (UniqueName: \"kubernetes.io/projected/a2cc16bb-ccb9-4852-aaa2-00a337a95ee2-kube-api-access-z4d4j\") pod \"opendatahub-operator-controller-manager-557b7f646c-84xtc\" (UID: \"a2cc16bb-ccb9-4852-aaa2-00a337a95ee2\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-84xtc" Jul 14 18:15:05.427263 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:05.427190 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-84xtc" Jul 14 18:15:05.565197 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:05.565162 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-557b7f646c-84xtc"] Jul 14 18:15:05.568911 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:15:05.568885 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2cc16bb_ccb9_4852_aaa2_00a337a95ee2.slice/crio-69f0431a24dcf34476cd252744d2fdf20721a9fdbcee53a47c43b89931c3e2e5 WatchSource:0}: Error finding container 69f0431a24dcf34476cd252744d2fdf20721a9fdbcee53a47c43b89931c3e2e5: Status 404 returned error can't find the container with id 69f0431a24dcf34476cd252744d2fdf20721a9fdbcee53a47c43b89931c3e2e5 Jul 14 18:15:05.570506 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:05.570491 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 18:15:05.948088 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:05.948057 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-84xtc" event={"ID":"a2cc16bb-ccb9-4852-aaa2-00a337a95ee2","Type":"ContainerStarted","Data":"69f0431a24dcf34476cd252744d2fdf20721a9fdbcee53a47c43b89931c3e2e5"} Jul 14 18:15:08.962856 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:08.962744 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-84xtc" event={"ID":"a2cc16bb-ccb9-4852-aaa2-00a337a95ee2","Type":"ContainerStarted","Data":"7e99a45f557ea4ccb3f0407dea0fbb648d58b5a15628466fa10379ced4c4dcb5"} Jul 14 18:15:09.968187 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:09.968154 2565 generic.go:356] "Generic (PLEG): container finished" podID="a2cc16bb-ccb9-4852-aaa2-00a337a95ee2" containerID="7e99a45f557ea4ccb3f0407dea0fbb648d58b5a15628466fa10379ced4c4dcb5" exitCode=0 Jul 14 18:15:09.968591 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:09.968247 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-84xtc" event={"ID":"a2cc16bb-ccb9-4852-aaa2-00a337a95ee2","Type":"ContainerDied","Data":"7e99a45f557ea4ccb3f0407dea0fbb648d58b5a15628466fa10379ced4c4dcb5"} Jul 14 18:15:10.975159 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:10.975129 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-84xtc" event={"ID":"a2cc16bb-ccb9-4852-aaa2-00a337a95ee2","Type":"ContainerStarted","Data":"08abe058474ee98ff270ef159388f8d357eacd1ad5034fe1245704bcf0257a5d"} Jul 14 18:15:10.975531 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:10.975268 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-84xtc" Jul 14 18:15:11.009394 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:11.009333 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-84xtc" podStartSLOduration=1.536604841 podStartE2EDuration="6.009311361s" totalImagesPullingTime="3.39102591s" totalInitContainerRuntime="1.08168061s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:15:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:15:11.005873403 +0000 UTC m=+315.843216657" watchObservedRunningTime="2026-07-14 18:15:11.009311361 +0000 UTC m=+315.846654619" Jul 14 18:15:21.981773 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:21.981740 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-84xtc" Jul 14 18:15:25.226270 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:25.226240 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wq6tl"] Jul 14 18:15:25.232141 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:25.232092 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-wq6tl" Jul 14 18:15:25.244759 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:25.244735 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhqw6\" (UniqueName: \"kubernetes.io/projected/8816c36a-81c8-4f59-8f8c-8d813e09d253-kube-api-access-bhqw6\") pod \"odh-model-controller-7998c67dc4-wq6tl\" (UID: \"8816c36a-81c8-4f59-8f8c-8d813e09d253\") " pod="opendatahub/odh-model-controller-7998c67dc4-wq6tl" Jul 14 18:15:25.244895 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:25.244780 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8816c36a-81c8-4f59-8f8c-8d813e09d253-cert\") pod \"odh-model-controller-7998c67dc4-wq6tl\" (UID: \"8816c36a-81c8-4f59-8f8c-8d813e09d253\") " pod="opendatahub/odh-model-controller-7998c67dc4-wq6tl" Jul 14 18:15:25.244895 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:25.244840 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-wg7zj\"" Jul 14 18:15:25.251841 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:25.251823 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 14 18:15:25.279418 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:25.279396 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wq6tl"] Jul 14 18:15:25.345252 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:25.345212 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8816c36a-81c8-4f59-8f8c-8d813e09d253-cert\") pod \"odh-model-controller-7998c67dc4-wq6tl\" (UID: \"8816c36a-81c8-4f59-8f8c-8d813e09d253\") " pod="opendatahub/odh-model-controller-7998c67dc4-wq6tl" Jul 14 18:15:25.345420 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:25.345298 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhqw6\" (UniqueName: \"kubernetes.io/projected/8816c36a-81c8-4f59-8f8c-8d813e09d253-kube-api-access-bhqw6\") pod \"odh-model-controller-7998c67dc4-wq6tl\" (UID: \"8816c36a-81c8-4f59-8f8c-8d813e09d253\") " pod="opendatahub/odh-model-controller-7998c67dc4-wq6tl" Jul 14 18:15:25.345420 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:15:25.345374 2565 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 14 18:15:25.345497 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:15:25.345440 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8816c36a-81c8-4f59-8f8c-8d813e09d253-cert podName:8816c36a-81c8-4f59-8f8c-8d813e09d253 nodeName:}" failed. No retries permitted until 2026-07-14 18:15:25.845421372 +0000 UTC m=+330.682764607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8816c36a-81c8-4f59-8f8c-8d813e09d253-cert") pod "odh-model-controller-7998c67dc4-wq6tl" (UID: "8816c36a-81c8-4f59-8f8c-8d813e09d253") : secret "odh-model-controller-webhook-cert" not found Jul 14 18:15:25.357486 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:25.357456 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhqw6\" (UniqueName: \"kubernetes.io/projected/8816c36a-81c8-4f59-8f8c-8d813e09d253-kube-api-access-bhqw6\") pod \"odh-model-controller-7998c67dc4-wq6tl\" (UID: \"8816c36a-81c8-4f59-8f8c-8d813e09d253\") " pod="opendatahub/odh-model-controller-7998c67dc4-wq6tl" Jul 14 18:15:25.849090 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:25.849060 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8816c36a-81c8-4f59-8f8c-8d813e09d253-cert\") pod \"odh-model-controller-7998c67dc4-wq6tl\" (UID: \"8816c36a-81c8-4f59-8f8c-8d813e09d253\") " pod="opendatahub/odh-model-controller-7998c67dc4-wq6tl" Jul 14 18:15:25.851462 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:25.851436 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8816c36a-81c8-4f59-8f8c-8d813e09d253-cert\") pod \"odh-model-controller-7998c67dc4-wq6tl\" (UID: \"8816c36a-81c8-4f59-8f8c-8d813e09d253\") " pod="opendatahub/odh-model-controller-7998c67dc4-wq6tl" Jul 14 18:15:26.142285 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:26.142190 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-wq6tl" Jul 14 18:15:26.285432 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:26.285398 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wq6tl"] Jul 14 18:15:26.289446 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:15:26.289406 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8816c36a_81c8_4f59_8f8c_8d813e09d253.slice/crio-7a0f4e2148221f48d5dedb92761e645f2a0b00844ac1cae9b1f594605dceb7fb WatchSource:0}: Error finding container 7a0f4e2148221f48d5dedb92761e645f2a0b00844ac1cae9b1f594605dceb7fb: Status 404 returned error can't find the container with id 7a0f4e2148221f48d5dedb92761e645f2a0b00844ac1cae9b1f594605dceb7fb Jul 14 18:15:27.040404 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:27.040373 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-wq6tl" event={"ID":"8816c36a-81c8-4f59-8f8c-8d813e09d253","Type":"ContainerStarted","Data":"7a0f4e2148221f48d5dedb92761e645f2a0b00844ac1cae9b1f594605dceb7fb"} Jul 14 18:15:30.056130 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:30.056018 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-wq6tl" event={"ID":"8816c36a-81c8-4f59-8f8c-8d813e09d253","Type":"ContainerStarted","Data":"d804ffd6387b43884d89b4c3c780e364c5dded16d1ef235659d7912fb9673667"} Jul 14 18:15:30.056588 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:30.056156 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-wq6tl" Jul 14 18:15:30.076016 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:30.075971 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-wq6tl" podStartSLOduration=1.582182012 podStartE2EDuration="5.075944815s" totalImagesPullingTime="3.493762803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:15:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:15:30.075024431 +0000 UTC m=+334.912367683" watchObservedRunningTime="2026-07-14 18:15:30.075944815 +0000 UTC m=+334.913288074" Jul 14 18:15:35.575457 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.575423 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-565f78b694-g6wdp"] Jul 14 18:15:35.578320 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.578299 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-565f78b694-g6wdp" Jul 14 18:15:35.581355 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.581337 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 14 18:15:35.581459 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.581337 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 14 18:15:35.597211 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.597190 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-565f78b694-g6wdp"] Jul 14 18:15:35.617811 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.617788 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vx6wz"] Jul 14 18:15:35.620882 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.620866 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vx6wz" Jul 14 18:15:35.623591 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.623563 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 14 18:15:35.623690 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.623625 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-tjzgn\"" Jul 14 18:15:35.626837 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.626640 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-494jk\" (UniqueName: \"kubernetes.io/projected/80e4139d-1310-414d-89c6-b775fad20eff-kube-api-access-494jk\") pod \"kube-auth-proxy-565f78b694-g6wdp\" (UID: \"80e4139d-1310-414d-89c6-b775fad20eff\") " pod="openshift-ingress/kube-auth-proxy-565f78b694-g6wdp" Jul 14 18:15:35.626837 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.626716 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80e4139d-1310-414d-89c6-b775fad20eff-tmp\") pod \"kube-auth-proxy-565f78b694-g6wdp\" (UID: \"80e4139d-1310-414d-89c6-b775fad20eff\") " pod="openshift-ingress/kube-auth-proxy-565f78b694-g6wdp" Jul 14 18:15:35.626837 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.626744 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80e4139d-1310-414d-89c6-b775fad20eff-tls-certs\") pod \"kube-auth-proxy-565f78b694-g6wdp\" (UID: \"80e4139d-1310-414d-89c6-b775fad20eff\") " pod="openshift-ingress/kube-auth-proxy-565f78b694-g6wdp" Jul 14 18:15:35.632653 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.632626 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vx6wz"] Jul 14 18:15:35.642453 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.642433 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-jhz62"] Jul 14 18:15:35.645756 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.645740 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jhz62" Jul 14 18:15:35.649186 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.649163 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-6zrj6\"" Jul 14 18:15:35.649277 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.649170 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 14 18:15:35.659117 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.659076 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-jhz62"] Jul 14 18:15:35.727800 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.727773 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ced6629e-359f-404f-9598-86ee1c9351a9-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vx6wz\" (UID: \"ced6629e-359f-404f-9598-86ee1c9351a9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vx6wz" Jul 14 18:15:35.727979 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.727807 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c94b2dd-bbdc-4b9b-a696-658cc441ed9e-cert\") pod \"kserve-controller-manager-d6f4b65b8-jhz62\" (UID: \"3c94b2dd-bbdc-4b9b-a696-658cc441ed9e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jhz62" Jul 14 18:15:35.727979 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.727845 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80e4139d-1310-414d-89c6-b775fad20eff-tmp\") pod \"kube-auth-proxy-565f78b694-g6wdp\" (UID: \"80e4139d-1310-414d-89c6-b775fad20eff\") " pod="openshift-ingress/kube-auth-proxy-565f78b694-g6wdp" Jul 14 18:15:35.727979 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.727871 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80e4139d-1310-414d-89c6-b775fad20eff-tls-certs\") pod \"kube-auth-proxy-565f78b694-g6wdp\" (UID: \"80e4139d-1310-414d-89c6-b775fad20eff\") " pod="openshift-ingress/kube-auth-proxy-565f78b694-g6wdp" Jul 14 18:15:35.727979 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.727895 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6gm5\" (UniqueName: \"kubernetes.io/projected/ced6629e-359f-404f-9598-86ee1c9351a9-kube-api-access-v6gm5\") pod \"llmisvc-controller-manager-777c6bd7d5-vx6wz\" (UID: \"ced6629e-359f-404f-9598-86ee1c9351a9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vx6wz" Jul 14 18:15:35.727979 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.727954 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vhfz\" (UniqueName: \"kubernetes.io/projected/3c94b2dd-bbdc-4b9b-a696-658cc441ed9e-kube-api-access-5vhfz\") pod \"kserve-controller-manager-d6f4b65b8-jhz62\" (UID: \"3c94b2dd-bbdc-4b9b-a696-658cc441ed9e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jhz62" Jul 14 18:15:35.728202 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.727985 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-494jk\" (UniqueName: \"kubernetes.io/projected/80e4139d-1310-414d-89c6-b775fad20eff-kube-api-access-494jk\") pod \"kube-auth-proxy-565f78b694-g6wdp\" (UID: \"80e4139d-1310-414d-89c6-b775fad20eff\") " pod="openshift-ingress/kube-auth-proxy-565f78b694-g6wdp" Jul 14 18:15:35.728202 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:15:35.728021 2565 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 14 18:15:35.728202 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:15:35.728093 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80e4139d-1310-414d-89c6-b775fad20eff-tls-certs podName:80e4139d-1310-414d-89c6-b775fad20eff nodeName:}" failed. No retries permitted until 2026-07-14 18:15:36.228073578 +0000 UTC m=+341.065416820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/80e4139d-1310-414d-89c6-b775fad20eff-tls-certs") pod "kube-auth-proxy-565f78b694-g6wdp" (UID: "80e4139d-1310-414d-89c6-b775fad20eff") : secret "kube-auth-proxy-tls" not found Jul 14 18:15:35.730267 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.730241 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80e4139d-1310-414d-89c6-b775fad20eff-tmp\") pod \"kube-auth-proxy-565f78b694-g6wdp\" (UID: \"80e4139d-1310-414d-89c6-b775fad20eff\") " pod="openshift-ingress/kube-auth-proxy-565f78b694-g6wdp" Jul 14 18:15:35.739898 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.739877 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-494jk\" (UniqueName: \"kubernetes.io/projected/80e4139d-1310-414d-89c6-b775fad20eff-kube-api-access-494jk\") pod \"kube-auth-proxy-565f78b694-g6wdp\" (UID: \"80e4139d-1310-414d-89c6-b775fad20eff\") " pod="openshift-ingress/kube-auth-proxy-565f78b694-g6wdp" Jul 14 18:15:35.828855 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.828772 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c94b2dd-bbdc-4b9b-a696-658cc441ed9e-cert\") pod \"kserve-controller-manager-d6f4b65b8-jhz62\" (UID: \"3c94b2dd-bbdc-4b9b-a696-658cc441ed9e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jhz62" Jul 14 18:15:35.828855 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.828831 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6gm5\" (UniqueName: \"kubernetes.io/projected/ced6629e-359f-404f-9598-86ee1c9351a9-kube-api-access-v6gm5\") pod \"llmisvc-controller-manager-777c6bd7d5-vx6wz\" (UID: \"ced6629e-359f-404f-9598-86ee1c9351a9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vx6wz" Jul 14 18:15:35.829053 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.828875 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vhfz\" (UniqueName: \"kubernetes.io/projected/3c94b2dd-bbdc-4b9b-a696-658cc441ed9e-kube-api-access-5vhfz\") pod \"kserve-controller-manager-d6f4b65b8-jhz62\" (UID: \"3c94b2dd-bbdc-4b9b-a696-658cc441ed9e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jhz62" Jul 14 18:15:35.829053 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.828926 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ced6629e-359f-404f-9598-86ee1c9351a9-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vx6wz\" (UID: \"ced6629e-359f-404f-9598-86ee1c9351a9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vx6wz" Jul 14 18:15:35.831509 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.831476 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ced6629e-359f-404f-9598-86ee1c9351a9-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vx6wz\" (UID: \"ced6629e-359f-404f-9598-86ee1c9351a9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vx6wz" Jul 14 18:15:35.831603 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.831519 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c94b2dd-bbdc-4b9b-a696-658cc441ed9e-cert\") pod \"kserve-controller-manager-d6f4b65b8-jhz62\" (UID: \"3c94b2dd-bbdc-4b9b-a696-658cc441ed9e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jhz62" Jul 14 18:15:35.845884 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.845852 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6gm5\" (UniqueName: \"kubernetes.io/projected/ced6629e-359f-404f-9598-86ee1c9351a9-kube-api-access-v6gm5\") pod \"llmisvc-controller-manager-777c6bd7d5-vx6wz\" (UID: \"ced6629e-359f-404f-9598-86ee1c9351a9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vx6wz" Jul 14 18:15:35.845987 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.845970 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vhfz\" (UniqueName: \"kubernetes.io/projected/3c94b2dd-bbdc-4b9b-a696-658cc441ed9e-kube-api-access-5vhfz\") pod \"kserve-controller-manager-d6f4b65b8-jhz62\" (UID: \"3c94b2dd-bbdc-4b9b-a696-658cc441ed9e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jhz62" Jul 14 18:15:35.930160 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.930129 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vx6wz" Jul 14 18:15:35.955318 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:35.955293 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jhz62" Jul 14 18:15:36.081578 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:36.081487 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vx6wz"] Jul 14 18:15:36.084683 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:15:36.084657 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podced6629e_359f_404f_9598_86ee1c9351a9.slice/crio-294b77bb6f0720deae12bd744ebb6665d019b1bca4905b59aa451da78062d2a7 WatchSource:0}: Error finding container 294b77bb6f0720deae12bd744ebb6665d019b1bca4905b59aa451da78062d2a7: Status 404 returned error can't find the container with id 294b77bb6f0720deae12bd744ebb6665d019b1bca4905b59aa451da78062d2a7 Jul 14 18:15:36.100691 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:36.100663 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-jhz62"] Jul 14 18:15:36.102940 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:15:36.102917 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c94b2dd_bbdc_4b9b_a696_658cc441ed9e.slice/crio-9a6c31855a56763bed223599a0294073193445b20ea034b87267a6f7867f15ee WatchSource:0}: Error finding container 9a6c31855a56763bed223599a0294073193445b20ea034b87267a6f7867f15ee: Status 404 returned error can't find the container with id 9a6c31855a56763bed223599a0294073193445b20ea034b87267a6f7867f15ee Jul 14 18:15:36.232962 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:36.232928 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80e4139d-1310-414d-89c6-b775fad20eff-tls-certs\") pod \"kube-auth-proxy-565f78b694-g6wdp\" (UID: \"80e4139d-1310-414d-89c6-b775fad20eff\") " pod="openshift-ingress/kube-auth-proxy-565f78b694-g6wdp" Jul 14 18:15:36.235551 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:36.235516 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80e4139d-1310-414d-89c6-b775fad20eff-tls-certs\") pod \"kube-auth-proxy-565f78b694-g6wdp\" (UID: \"80e4139d-1310-414d-89c6-b775fad20eff\") " pod="openshift-ingress/kube-auth-proxy-565f78b694-g6wdp" Jul 14 18:15:36.488151 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:36.488061 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-565f78b694-g6wdp" Jul 14 18:15:36.629489 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:36.629467 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-565f78b694-g6wdp"] Jul 14 18:15:36.631680 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:15:36.631649 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e4139d_1310_414d_89c6_b775fad20eff.slice/crio-556a9c7babbae15a05766269d393c73a5d5291218f06359340de1356bd555bfb WatchSource:0}: Error finding container 556a9c7babbae15a05766269d393c73a5d5291218f06359340de1356bd555bfb: Status 404 returned error can't find the container with id 556a9c7babbae15a05766269d393c73a5d5291218f06359340de1356bd555bfb Jul 14 18:15:37.087891 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.087850 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-565f78b694-g6wdp" event={"ID":"80e4139d-1310-414d-89c6-b775fad20eff","Type":"ContainerStarted","Data":"556a9c7babbae15a05766269d393c73a5d5291218f06359340de1356bd555bfb"} Jul 14 18:15:37.089368 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.089340 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jhz62" event={"ID":"3c94b2dd-bbdc-4b9b-a696-658cc441ed9e","Type":"ContainerStarted","Data":"9a6c31855a56763bed223599a0294073193445b20ea034b87267a6f7867f15ee"} Jul 14 18:15:37.090554 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.090454 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vx6wz" event={"ID":"ced6629e-359f-404f-9598-86ee1c9351a9","Type":"ContainerStarted","Data":"294b77bb6f0720deae12bd744ebb6665d019b1bca4905b59aa451da78062d2a7"} Jul 14 18:15:37.444898 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.444809 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch"] Jul 14 18:15:37.448048 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.448027 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch" Jul 14 18:15:37.459768 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.454126 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-nppx4\"" Jul 14 18:15:37.459768 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.454614 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 14 18:15:37.459768 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.454830 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 14 18:15:37.459768 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.454879 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 14 18:15:37.459768 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.454906 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 14 18:15:37.472648 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.472621 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch"] Jul 14 18:15:37.548605 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.548573 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45ec5309-1cfc-4477-aa0f-082ce7485a9c-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-npbch\" (UID: \"45ec5309-1cfc-4477-aa0f-082ce7485a9c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch" Jul 14 18:15:37.548605 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.548604 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/45ec5309-1cfc-4477-aa0f-082ce7485a9c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-npbch\" (UID: \"45ec5309-1cfc-4477-aa0f-082ce7485a9c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch" Jul 14 18:15:37.548792 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.548624 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/45ec5309-1cfc-4477-aa0f-082ce7485a9c-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-npbch\" (UID: \"45ec5309-1cfc-4477-aa0f-082ce7485a9c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch" Jul 14 18:15:37.548792 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.548639 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/45ec5309-1cfc-4477-aa0f-082ce7485a9c-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-npbch\" (UID: \"45ec5309-1cfc-4477-aa0f-082ce7485a9c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch" Jul 14 18:15:37.548792 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.548715 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtksp\" (UniqueName: \"kubernetes.io/projected/45ec5309-1cfc-4477-aa0f-082ce7485a9c-kube-api-access-xtksp\") pod \"istiod-openshift-gateway-94bb8fbfd-npbch\" (UID: \"45ec5309-1cfc-4477-aa0f-082ce7485a9c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch" Jul 14 18:15:37.548792 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.548749 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/45ec5309-1cfc-4477-aa0f-082ce7485a9c-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-npbch\" (UID: \"45ec5309-1cfc-4477-aa0f-082ce7485a9c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch" Jul 14 18:15:37.548792 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.548768 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/45ec5309-1cfc-4477-aa0f-082ce7485a9c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-npbch\" (UID: \"45ec5309-1cfc-4477-aa0f-082ce7485a9c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch" Jul 14 18:15:37.649605 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.649571 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45ec5309-1cfc-4477-aa0f-082ce7485a9c-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-npbch\" (UID: \"45ec5309-1cfc-4477-aa0f-082ce7485a9c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch" Jul 14 18:15:37.649605 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.649605 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/45ec5309-1cfc-4477-aa0f-082ce7485a9c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-npbch\" (UID: \"45ec5309-1cfc-4477-aa0f-082ce7485a9c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch" Jul 14 18:15:37.650093 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.649626 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/45ec5309-1cfc-4477-aa0f-082ce7485a9c-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-npbch\" (UID: \"45ec5309-1cfc-4477-aa0f-082ce7485a9c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch" Jul 14 18:15:37.650093 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.649643 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/45ec5309-1cfc-4477-aa0f-082ce7485a9c-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-npbch\" (UID: \"45ec5309-1cfc-4477-aa0f-082ce7485a9c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch" Jul 14 18:15:37.650093 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.649675 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtksp\" (UniqueName: \"kubernetes.io/projected/45ec5309-1cfc-4477-aa0f-082ce7485a9c-kube-api-access-xtksp\") pod \"istiod-openshift-gateway-94bb8fbfd-npbch\" (UID: \"45ec5309-1cfc-4477-aa0f-082ce7485a9c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch" Jul 14 18:15:37.650093 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.649706 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/45ec5309-1cfc-4477-aa0f-082ce7485a9c-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-npbch\" (UID: \"45ec5309-1cfc-4477-aa0f-082ce7485a9c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch" Jul 14 18:15:37.650093 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.649729 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/45ec5309-1cfc-4477-aa0f-082ce7485a9c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-npbch\" (UID: \"45ec5309-1cfc-4477-aa0f-082ce7485a9c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch" Jul 14 18:15:37.650356 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.650264 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/45ec5309-1cfc-4477-aa0f-082ce7485a9c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-npbch\" (UID: \"45ec5309-1cfc-4477-aa0f-082ce7485a9c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch" Jul 14 18:15:37.652374 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.652341 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/45ec5309-1cfc-4477-aa0f-082ce7485a9c-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-npbch\" (UID: \"45ec5309-1cfc-4477-aa0f-082ce7485a9c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch" Jul 14 18:15:37.652507 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.652386 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/45ec5309-1cfc-4477-aa0f-082ce7485a9c-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-npbch\" (UID: \"45ec5309-1cfc-4477-aa0f-082ce7485a9c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch" Jul 14 18:15:37.652507 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.652446 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/45ec5309-1cfc-4477-aa0f-082ce7485a9c-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-npbch\" (UID: \"45ec5309-1cfc-4477-aa0f-082ce7485a9c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch" Jul 14 18:15:37.652507 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.652488 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/45ec5309-1cfc-4477-aa0f-082ce7485a9c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-npbch\" (UID: \"45ec5309-1cfc-4477-aa0f-082ce7485a9c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch" Jul 14 18:15:37.663145 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.663121 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45ec5309-1cfc-4477-aa0f-082ce7485a9c-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-npbch\" (UID: \"45ec5309-1cfc-4477-aa0f-082ce7485a9c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch" Jul 14 18:15:37.671560 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.671539 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtksp\" (UniqueName: \"kubernetes.io/projected/45ec5309-1cfc-4477-aa0f-082ce7485a9c-kube-api-access-xtksp\") pod \"istiod-openshift-gateway-94bb8fbfd-npbch\" (UID: \"45ec5309-1cfc-4477-aa0f-082ce7485a9c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch" Jul 14 18:15:37.763134 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.763050 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch" Jul 14 18:15:37.926485 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:37.926457 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch"] Jul 14 18:15:37.927990 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:15:37.927960 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ec5309_1cfc_4477_aa0f_082ce7485a9c.slice/crio-05d329a13c984f415a6d6b3bbcc86c2e775f32954c64f0935fb62839e835e9ac WatchSource:0}: Error finding container 05d329a13c984f415a6d6b3bbcc86c2e775f32954c64f0935fb62839e835e9ac: Status 404 returned error can't find the container with id 05d329a13c984f415a6d6b3bbcc86c2e775f32954c64f0935fb62839e835e9ac Jul 14 18:15:38.096542 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:38.096509 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch" event={"ID":"45ec5309-1cfc-4477-aa0f-082ce7485a9c","Type":"ContainerStarted","Data":"05d329a13c984f415a6d6b3bbcc86c2e775f32954c64f0935fb62839e835e9ac"} Jul 14 18:15:40.347155 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:40.347118 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:15:40.347433 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:40.347190 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:15:41.063218 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:41.063185 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-wq6tl" Jul 14 18:15:41.113677 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:41.113489 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch" event={"ID":"45ec5309-1cfc-4477-aa0f-082ce7485a9c","Type":"ContainerStarted","Data":"813ba326f2f4de72d3f34db061b941c8014064a827a52e0861e65b0f55543533"} Jul 14 18:15:41.113848 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:41.113690 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch" Jul 14 18:15:41.115210 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:41.114903 2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-npbch container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 18:15:41.115210 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:41.114962 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch" podUID="45ec5309-1cfc-4477-aa0f-082ce7485a9c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 18:15:41.137962 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:41.136785 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch" podStartSLOduration=1.720018859 podStartE2EDuration="4.136766394s" totalImagesPullingTime="2.416747535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:15:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:15:41.136217162 +0000 UTC m=+345.973560434" watchObservedRunningTime="2026-07-14 18:15:41.136766394 +0000 UTC m=+345.974109651" Jul 14 18:15:42.118850 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:42.118789 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jhz62" event={"ID":"3c94b2dd-bbdc-4b9b-a696-658cc441ed9e","Type":"ContainerStarted","Data":"b91cd5e076cd24de8aff12b6daeca7ca767a1c73d7a2c02d7c98b205fbb1c45a"} Jul 14 18:15:42.119575 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:42.119552 2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-npbch container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 18:15:42.119666 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:42.119601 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch" podUID="45ec5309-1cfc-4477-aa0f-082ce7485a9c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 18:15:42.141118 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:42.141061 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jhz62" podStartSLOduration=1.82774039 podStartE2EDuration="7.141044337s" totalImagesPullingTime="5.313303947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:15:42.140750065 +0000 UTC m=+346.978093342" watchObservedRunningTime="2026-07-14 18:15:42.141044337 +0000 UTC m=+346.978387595" Jul 14 18:15:43.123937 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:43.123908 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jhz62" Jul 14 18:15:44.129818 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:44.129778 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-565f78b694-g6wdp" event={"ID":"80e4139d-1310-414d-89c6-b775fad20eff","Type":"ContainerStarted","Data":"2eb571e4c562944c226957126cafde5b7950750c365afefab4f5e547cf81e72a"} Jul 14 18:15:44.158333 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:44.158283 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-565f78b694-g6wdp" podStartSLOduration=2.609663931 podStartE2EDuration="9.158264337s" totalImagesPullingTime="6.548600406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:15:44.156433166 +0000 UTC m=+348.993776424" watchObservedRunningTime="2026-07-14 18:15:44.158264337 +0000 UTC m=+348.995607593" Jul 14 18:15:45.119982 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:45.119952 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-npbch" Jul 14 18:15:45.134998 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:45.134969 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vx6wz" event={"ID":"ced6629e-359f-404f-9598-86ee1c9351a9","Type":"ContainerStarted","Data":"ed3c955221c434ec85ff2b0d6ce041102a5d9668461898480780e55c1d4d367f"} Jul 14 18:15:45.135363 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:45.135136 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vx6wz" Jul 14 18:15:45.163032 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:15:45.162978 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vx6wz" podStartSLOduration=1.540066127 podStartE2EDuration="10.162959364s" totalImagesPullingTime="8.622893237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:15:45.162015419 +0000 UTC m=+349.999358913" watchObservedRunningTime="2026-07-14 18:15:45.162959364 +0000 UTC m=+350.000302621" Jul 14 18:16:14.136762 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:16:14.136723 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jhz62" Jul 14 18:16:16.140697 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:16:16.140666 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vx6wz" Jul 14 18:17:10.443084 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:10.443047 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zgvrp"] Jul 14 18:17:10.445231 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:10.445215 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zgvrp" Jul 14 18:17:10.452321 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:10.452301 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-k5cr2\"" Jul 14 18:17:10.452425 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:10.452393 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 14 18:17:10.452469 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:10.452444 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 14 18:17:10.464488 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:10.464464 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zgvrp"] Jul 14 18:17:10.579664 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:10.579637 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a152f968-c3bc-4158-8bab-9154df6c8c70-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zgvrp\" (UID: \"a152f968-c3bc-4158-8bab-9154df6c8c70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zgvrp" Jul 14 18:17:10.579828 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:10.579673 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdfp7\" (UniqueName: \"kubernetes.io/projected/a152f968-c3bc-4158-8bab-9154df6c8c70-kube-api-access-vdfp7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zgvrp\" (UID: \"a152f968-c3bc-4158-8bab-9154df6c8c70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zgvrp" Jul 14 18:17:10.680983 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:10.680955 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a152f968-c3bc-4158-8bab-9154df6c8c70-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zgvrp\" (UID: \"a152f968-c3bc-4158-8bab-9154df6c8c70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zgvrp" Jul 14 18:17:10.681184 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:10.680990 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdfp7\" (UniqueName: \"kubernetes.io/projected/a152f968-c3bc-4158-8bab-9154df6c8c70-kube-api-access-vdfp7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zgvrp\" (UID: \"a152f968-c3bc-4158-8bab-9154df6c8c70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zgvrp" Jul 14 18:17:10.681385 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:10.681365 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a152f968-c3bc-4158-8bab-9154df6c8c70-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zgvrp\" (UID: \"a152f968-c3bc-4158-8bab-9154df6c8c70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zgvrp" Jul 14 18:17:10.691739 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:10.691718 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdfp7\" (UniqueName: \"kubernetes.io/projected/a152f968-c3bc-4158-8bab-9154df6c8c70-kube-api-access-vdfp7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zgvrp\" (UID: \"a152f968-c3bc-4158-8bab-9154df6c8c70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zgvrp" Jul 14 18:17:10.755269 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:10.755213 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zgvrp" Jul 14 18:17:10.888592 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:10.888548 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zgvrp"] Jul 14 18:17:10.890982 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:17:10.890957 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda152f968_c3bc_4158_8bab_9154df6c8c70.slice/crio-54b5b7aa9520a394e233c3a862e60ca14dae34ea45aeaa35e1a1f56c480d94bc WatchSource:0}: Error finding container 54b5b7aa9520a394e233c3a862e60ca14dae34ea45aeaa35e1a1f56c480d94bc: Status 404 returned error can't find the container with id 54b5b7aa9520a394e233c3a862e60ca14dae34ea45aeaa35e1a1f56c480d94bc Jul 14 18:17:11.496613 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:11.496582 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zgvrp" event={"ID":"a152f968-c3bc-4158-8bab-9154df6c8c70","Type":"ContainerStarted","Data":"54b5b7aa9520a394e233c3a862e60ca14dae34ea45aeaa35e1a1f56c480d94bc"} Jul 14 18:17:18.529957 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:18.529923 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zgvrp" event={"ID":"a152f968-c3bc-4158-8bab-9154df6c8c70","Type":"ContainerStarted","Data":"ebdf8ed4de45d39a03f280ff97ced27ccfd44648d78a56392ac77aed0cfb9fb5"} Jul 14 18:17:18.530351 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:18.530060 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zgvrp" Jul 14 18:17:18.560584 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:18.560539 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zgvrp" podStartSLOduration=1.228680188 podStartE2EDuration="8.560523728s" totalImagesPullingTime="7.33184354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:17:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:17:18.557945543 +0000 UTC m=+443.395288799" watchObservedRunningTime="2026-07-14 18:17:18.560523728 +0000 UTC m=+443.397867049" Jul 14 18:17:29.536778 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:29.536740 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zgvrp" Jul 14 18:17:30.824550 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:30.824504 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zh598"] Jul 14 18:17:30.827993 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:30.827971 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zh598" Jul 14 18:17:30.840052 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:30.840027 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zh598"] Jul 14 18:17:30.953338 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:30.953297 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpb6m\" (UniqueName: \"kubernetes.io/projected/d9e62ddc-5d97-4519-86a3-dc69135a97d3-kube-api-access-lpb6m\") pod \"kuadrant-operator-controller-manager-55467d9f7-zh598\" (UID: \"d9e62ddc-5d97-4519-86a3-dc69135a97d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zh598" Jul 14 18:17:30.953483 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:30.953378 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9e62ddc-5d97-4519-86a3-dc69135a97d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-zh598\" (UID: \"d9e62ddc-5d97-4519-86a3-dc69135a97d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zh598" Jul 14 18:17:31.054719 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.054682 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpb6m\" (UniqueName: \"kubernetes.io/projected/d9e62ddc-5d97-4519-86a3-dc69135a97d3-kube-api-access-lpb6m\") pod \"kuadrant-operator-controller-manager-55467d9f7-zh598\" (UID: \"d9e62ddc-5d97-4519-86a3-dc69135a97d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zh598" Jul 14 18:17:31.054865 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.054762 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9e62ddc-5d97-4519-86a3-dc69135a97d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-zh598\" (UID: \"d9e62ddc-5d97-4519-86a3-dc69135a97d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zh598" Jul 14 18:17:31.055142 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.055127 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9e62ddc-5d97-4519-86a3-dc69135a97d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-zh598\" (UID: \"d9e62ddc-5d97-4519-86a3-dc69135a97d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zh598" Jul 14 18:17:31.064362 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.064338 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpb6m\" (UniqueName: \"kubernetes.io/projected/d9e62ddc-5d97-4519-86a3-dc69135a97d3-kube-api-access-lpb6m\") pod \"kuadrant-operator-controller-manager-55467d9f7-zh598\" (UID: \"d9e62ddc-5d97-4519-86a3-dc69135a97d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zh598" Jul 14 18:17:31.138331 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.138239 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zh598" Jul 14 18:17:31.281447 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.281421 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zh598"] Jul 14 18:17:31.284519 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:17:31.284492 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e62ddc_5d97_4519_86a3_dc69135a97d3.slice/crio-3efba517228ca237bb2e7f771e9ffa80b5125e2bc187064c58f118e99c3235c7 WatchSource:0}: Error finding container 3efba517228ca237bb2e7f771e9ffa80b5125e2bc187064c58f118e99c3235c7: Status 404 returned error can't find the container with id 3efba517228ca237bb2e7f771e9ffa80b5125e2bc187064c58f118e99c3235c7 Jul 14 18:17:31.297737 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.297713 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zh598"] Jul 14 18:17:31.306070 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.306046 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zh598"] Jul 14 18:17:31.321461 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.321434 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rfzgp"] Jul 14 18:17:31.325927 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.325906 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rfzgp" Jul 14 18:17:31.338215 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.338190 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zgvrp"] Jul 14 18:17:31.338554 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.338530 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zgvrp" podUID="a152f968-c3bc-4158-8bab-9154df6c8c70" containerName="manager" containerID="cri-o://ebdf8ed4de45d39a03f280ff97ced27ccfd44648d78a56392ac77aed0cfb9fb5" gracePeriod=2 Jul 14 18:17:31.340793 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.340772 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rfzgp"] Jul 14 18:17:31.351520 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.351049 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zgvrp"] Jul 14 18:17:31.355488 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:17:31.355180 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zgvrp\" is forbidden: User \"system:node:ip-10-0-136-25.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-25.ec2.internal' and this object" podUID="a152f968-c3bc-4158-8bab-9154df6c8c70" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zgvrp" Jul 14 18:17:31.357194 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.357167 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a1aa1c0-0912-47da-9270-cb270d0cbc24-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-rfzgp\" (UID: \"4a1aa1c0-0912-47da-9270-cb270d0cbc24\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rfzgp" Jul 14 18:17:31.357294 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.357212 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqr22\" (UniqueName: \"kubernetes.io/projected/4a1aa1c0-0912-47da-9270-cb270d0cbc24-kube-api-access-rqr22\") pod \"kuadrant-operator-controller-manager-55467d9f7-rfzgp\" (UID: \"4a1aa1c0-0912-47da-9270-cb270d0cbc24\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rfzgp" Jul 14 18:17:31.457875 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.457787 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a1aa1c0-0912-47da-9270-cb270d0cbc24-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-rfzgp\" (UID: \"4a1aa1c0-0912-47da-9270-cb270d0cbc24\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rfzgp" Jul 14 18:17:31.457875 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.457831 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqr22\" (UniqueName: \"kubernetes.io/projected/4a1aa1c0-0912-47da-9270-cb270d0cbc24-kube-api-access-rqr22\") pod \"kuadrant-operator-controller-manager-55467d9f7-rfzgp\" (UID: \"4a1aa1c0-0912-47da-9270-cb270d0cbc24\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rfzgp" Jul 14 18:17:31.458195 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.458172 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a1aa1c0-0912-47da-9270-cb270d0cbc24-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-rfzgp\" (UID: \"4a1aa1c0-0912-47da-9270-cb270d0cbc24\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rfzgp" Jul 14 18:17:31.470226 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.470195 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqr22\" (UniqueName: \"kubernetes.io/projected/4a1aa1c0-0912-47da-9270-cb270d0cbc24-kube-api-access-rqr22\") pod \"kuadrant-operator-controller-manager-55467d9f7-rfzgp\" (UID: \"4a1aa1c0-0912-47da-9270-cb270d0cbc24\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rfzgp" Jul 14 18:17:31.578718 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.578690 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zgvrp" Jul 14 18:17:31.581294 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:17:31.581267 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zgvrp\" is forbidden: User \"system:node:ip-10-0-136-25.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-25.ec2.internal' and this object" podUID="a152f968-c3bc-4158-8bab-9154df6c8c70" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zgvrp" Jul 14 18:17:31.585523 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.585493 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zh598" podUID="d9e62ddc-5d97-4519-86a3-dc69135a97d3" containerName="manager" containerID="cri-o://fc0bbb91b4f9358c4b400fe50df09e93c0361b9846f7142d69d19c07dc0a0c0b" gracePeriod=2 Jul 14 18:17:31.587027 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.587003 2565 generic.go:356] "Generic (PLEG): container finished" podID="a152f968-c3bc-4158-8bab-9154df6c8c70" containerID="ebdf8ed4de45d39a03f280ff97ced27ccfd44648d78a56392ac77aed0cfb9fb5" exitCode=0 Jul 14 18:17:31.587143 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.587043 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zgvrp" Jul 14 18:17:31.587143 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.587079 2565 scope.go:117] "RemoveContainer" containerID="ebdf8ed4de45d39a03f280ff97ced27ccfd44648d78a56392ac77aed0cfb9fb5" Jul 14 18:17:31.588264 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:17:31.588237 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zgvrp\" is forbidden: User \"system:node:ip-10-0-136-25.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-25.ec2.internal' and this object" podUID="a152f968-c3bc-4158-8bab-9154df6c8c70" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zgvrp" Jul 14 18:17:31.590465 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:17:31.590445 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-zh598\" is forbidden: User \"system:node:ip-10-0-136-25.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-25.ec2.internal' and this object" podUID="d9e62ddc-5d97-4519-86a3-dc69135a97d3" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zh598" Jul 14 18:17:31.592956 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:17:31.592935 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zgvrp\" is forbidden: User \"system:node:ip-10-0-136-25.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-25.ec2.internal' and this object" podUID="a152f968-c3bc-4158-8bab-9154df6c8c70" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zgvrp" Jul 14 18:17:31.595074 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:17:31.595025 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-zh598\" is forbidden: User \"system:node:ip-10-0-136-25.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-25.ec2.internal' and this object" podUID="d9e62ddc-5d97-4519-86a3-dc69135a97d3" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zh598" Jul 14 18:17:31.596476 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.596456 2565 scope.go:117] "RemoveContainer" containerID="ebdf8ed4de45d39a03f280ff97ced27ccfd44648d78a56392ac77aed0cfb9fb5" Jul 14 18:17:31.596785 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:17:31.596762 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebdf8ed4de45d39a03f280ff97ced27ccfd44648d78a56392ac77aed0cfb9fb5\": container with ID starting with ebdf8ed4de45d39a03f280ff97ced27ccfd44648d78a56392ac77aed0cfb9fb5 not found: ID does not exist" containerID="ebdf8ed4de45d39a03f280ff97ced27ccfd44648d78a56392ac77aed0cfb9fb5" Jul 14 18:17:31.596848 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.596797 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebdf8ed4de45d39a03f280ff97ced27ccfd44648d78a56392ac77aed0cfb9fb5"} err="failed to get container status \"ebdf8ed4de45d39a03f280ff97ced27ccfd44648d78a56392ac77aed0cfb9fb5\": rpc error: code = NotFound desc = could not find container \"ebdf8ed4de45d39a03f280ff97ced27ccfd44648d78a56392ac77aed0cfb9fb5\": container with ID starting with ebdf8ed4de45d39a03f280ff97ced27ccfd44648d78a56392ac77aed0cfb9fb5 not found: ID does not exist" Jul 14 18:17:31.641328 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.641299 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rfzgp" Jul 14 18:17:31.659604 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.659580 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a152f968-c3bc-4158-8bab-9154df6c8c70-extensions-socket-volume\") pod \"a152f968-c3bc-4158-8bab-9154df6c8c70\" (UID: \"a152f968-c3bc-4158-8bab-9154df6c8c70\") " Jul 14 18:17:31.659734 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.659620 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdfp7\" (UniqueName: \"kubernetes.io/projected/a152f968-c3bc-4158-8bab-9154df6c8c70-kube-api-access-vdfp7\") pod \"a152f968-c3bc-4158-8bab-9154df6c8c70\" (UID: \"a152f968-c3bc-4158-8bab-9154df6c8c70\") " Jul 14 18:17:31.660165 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.660137 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a152f968-c3bc-4158-8bab-9154df6c8c70-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a152f968-c3bc-4158-8bab-9154df6c8c70" (UID: "a152f968-c3bc-4158-8bab-9154df6c8c70"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:17:31.661896 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.661877 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a152f968-c3bc-4158-8bab-9154df6c8c70-kube-api-access-vdfp7" (OuterVolumeSpecName: "kube-api-access-vdfp7") pod "a152f968-c3bc-4158-8bab-9154df6c8c70" (UID: "a152f968-c3bc-4158-8bab-9154df6c8c70"). InnerVolumeSpecName "kube-api-access-vdfp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:17:31.690398 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.690367 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a152f968-c3bc-4158-8bab-9154df6c8c70" path="/var/lib/kubelet/pods/a152f968-c3bc-4158-8bab-9154df6c8c70/volumes" Jul 14 18:17:31.761253 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.760948 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdfp7\" (UniqueName: \"kubernetes.io/projected/a152f968-c3bc-4158-8bab-9154df6c8c70-kube-api-access-vdfp7\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:17:31.761253 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.760980 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a152f968-c3bc-4158-8bab-9154df6c8c70-extensions-socket-volume\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:17:31.801740 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.801704 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rfzgp"] Jul 14 18:17:31.805997 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:17:31.805968 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a1aa1c0_0912_47da_9270_cb270d0cbc24.slice/crio-51a3900898bbdda4077eece4490c90dd1d711bfa97b1d0dcd5afba673551638a WatchSource:0}: Error finding container 51a3900898bbdda4077eece4490c90dd1d711bfa97b1d0dcd5afba673551638a: Status 404 returned error can't find the container with id 51a3900898bbdda4077eece4490c90dd1d711bfa97b1d0dcd5afba673551638a Jul 14 18:17:31.832801 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.832778 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zh598" Jul 14 18:17:31.861396 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.861367 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9e62ddc-5d97-4519-86a3-dc69135a97d3-extensions-socket-volume\") pod \"d9e62ddc-5d97-4519-86a3-dc69135a97d3\" (UID: \"d9e62ddc-5d97-4519-86a3-dc69135a97d3\") " Jul 14 18:17:31.861557 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.861429 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpb6m\" (UniqueName: \"kubernetes.io/projected/d9e62ddc-5d97-4519-86a3-dc69135a97d3-kube-api-access-lpb6m\") pod \"d9e62ddc-5d97-4519-86a3-dc69135a97d3\" (UID: \"d9e62ddc-5d97-4519-86a3-dc69135a97d3\") " Jul 14 18:17:31.861614 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.861589 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e62ddc-5d97-4519-86a3-dc69135a97d3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d9e62ddc-5d97-4519-86a3-dc69135a97d3" (UID: "d9e62ddc-5d97-4519-86a3-dc69135a97d3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:17:31.863740 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.863716 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e62ddc-5d97-4519-86a3-dc69135a97d3-kube-api-access-lpb6m" (OuterVolumeSpecName: "kube-api-access-lpb6m") pod "d9e62ddc-5d97-4519-86a3-dc69135a97d3" (UID: "d9e62ddc-5d97-4519-86a3-dc69135a97d3"). InnerVolumeSpecName "kube-api-access-lpb6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:17:31.962079 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.962041 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9e62ddc-5d97-4519-86a3-dc69135a97d3-extensions-socket-volume\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:17:31.962079 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:31.962071 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpb6m\" (UniqueName: \"kubernetes.io/projected/d9e62ddc-5d97-4519-86a3-dc69135a97d3-kube-api-access-lpb6m\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:17:32.593892 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:32.593850 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rfzgp" event={"ID":"4a1aa1c0-0912-47da-9270-cb270d0cbc24","Type":"ContainerStarted","Data":"f4fbdf5351090de1dd56f45a1b4198f9a62c7c6aa95922ec8fe2ca21d3768a1b"} Jul 14 18:17:32.594119 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:32.593918 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rfzgp" event={"ID":"4a1aa1c0-0912-47da-9270-cb270d0cbc24","Type":"ContainerStarted","Data":"51a3900898bbdda4077eece4490c90dd1d711bfa97b1d0dcd5afba673551638a"} Jul 14 18:17:32.594119 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:32.594020 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rfzgp" Jul 14 18:17:32.595443 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:32.595419 2565 generic.go:356] "Generic (PLEG): container finished" podID="d9e62ddc-5d97-4519-86a3-dc69135a97d3" containerID="fc0bbb91b4f9358c4b400fe50df09e93c0361b9846f7142d69d19c07dc0a0c0b" exitCode=2 Jul 14 18:17:32.595577 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:32.595460 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zh598" Jul 14 18:17:32.595577 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:32.595513 2565 scope.go:117] "RemoveContainer" containerID="fc0bbb91b4f9358c4b400fe50df09e93c0361b9846f7142d69d19c07dc0a0c0b" Jul 14 18:17:32.604518 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:32.604499 2565 scope.go:117] "RemoveContainer" containerID="fc0bbb91b4f9358c4b400fe50df09e93c0361b9846f7142d69d19c07dc0a0c0b" Jul 14 18:17:32.604791 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:17:32.604775 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc0bbb91b4f9358c4b400fe50df09e93c0361b9846f7142d69d19c07dc0a0c0b\": container with ID starting with fc0bbb91b4f9358c4b400fe50df09e93c0361b9846f7142d69d19c07dc0a0c0b not found: ID does not exist" containerID="fc0bbb91b4f9358c4b400fe50df09e93c0361b9846f7142d69d19c07dc0a0c0b" Jul 14 18:17:32.604840 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:32.604800 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc0bbb91b4f9358c4b400fe50df09e93c0361b9846f7142d69d19c07dc0a0c0b"} err="failed to get container status \"fc0bbb91b4f9358c4b400fe50df09e93c0361b9846f7142d69d19c07dc0a0c0b\": rpc error: code = NotFound desc = could not find container \"fc0bbb91b4f9358c4b400fe50df09e93c0361b9846f7142d69d19c07dc0a0c0b\": container with ID starting with fc0bbb91b4f9358c4b400fe50df09e93c0361b9846f7142d69d19c07dc0a0c0b not found: ID does not exist" Jul 14 18:17:32.629991 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:32.629944 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rfzgp" podStartSLOduration=1.629928971 podStartE2EDuration="1.629928971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:17:32.626956382 +0000 UTC m=+457.464299641" watchObservedRunningTime="2026-07-14 18:17:32.629928971 +0000 UTC m=+457.467272229" Jul 14 18:17:32.635558 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:17:32.635530 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-zh598\" is forbidden: User \"system:node:ip-10-0-136-25.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-25.ec2.internal' and this object" podUID="d9e62ddc-5d97-4519-86a3-dc69135a97d3" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zh598" Jul 14 18:17:33.689173 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:33.689146 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9e62ddc-5d97-4519-86a3-dc69135a97d3" path="/var/lib/kubelet/pods/d9e62ddc-5d97-4519-86a3-dc69135a97d3/volumes" Jul 14 18:17:43.605022 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:43.604988 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rfzgp" Jul 14 18:17:43.960533 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:43.960501 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-957w7"] Jul 14 18:17:43.960842 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:43.960829 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a152f968-c3bc-4158-8bab-9154df6c8c70" containerName="manager" Jul 14 18:17:43.960896 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:43.960845 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a152f968-c3bc-4158-8bab-9154df6c8c70" containerName="manager" Jul 14 18:17:43.960896 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:43.960857 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9e62ddc-5d97-4519-86a3-dc69135a97d3" containerName="manager" Jul 14 18:17:43.960896 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:43.960863 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9e62ddc-5d97-4519-86a3-dc69135a97d3" containerName="manager" Jul 14 18:17:43.961004 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:43.960917 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9e62ddc-5d97-4519-86a3-dc69135a97d3" containerName="manager" Jul 14 18:17:43.961004 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:43.960926 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a152f968-c3bc-4158-8bab-9154df6c8c70" containerName="manager" Jul 14 18:17:43.964870 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:43.964830 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-957w7" Jul 14 18:17:43.977440 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:43.977409 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-957w7"] Jul 14 18:17:44.055630 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:44.055591 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/357ff5a6-97ce-450b-9948-94c31be65cfe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-957w7\" (UID: \"357ff5a6-97ce-450b-9948-94c31be65cfe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-957w7" Jul 14 18:17:44.055630 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:44.055635 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cccgh\" (UniqueName: \"kubernetes.io/projected/357ff5a6-97ce-450b-9948-94c31be65cfe-kube-api-access-cccgh\") pod \"kuadrant-operator-controller-manager-86878c769f-957w7\" (UID: \"357ff5a6-97ce-450b-9948-94c31be65cfe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-957w7" Jul 14 18:17:44.156822 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:44.156790 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/357ff5a6-97ce-450b-9948-94c31be65cfe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-957w7\" (UID: \"357ff5a6-97ce-450b-9948-94c31be65cfe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-957w7" Jul 14 18:17:44.156822 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:44.156828 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cccgh\" (UniqueName: \"kubernetes.io/projected/357ff5a6-97ce-450b-9948-94c31be65cfe-kube-api-access-cccgh\") pod \"kuadrant-operator-controller-manager-86878c769f-957w7\" (UID: \"357ff5a6-97ce-450b-9948-94c31be65cfe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-957w7" Jul 14 18:17:44.157212 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:44.157196 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/357ff5a6-97ce-450b-9948-94c31be65cfe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-957w7\" (UID: \"357ff5a6-97ce-450b-9948-94c31be65cfe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-957w7" Jul 14 18:17:44.170618 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:44.170589 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cccgh\" (UniqueName: \"kubernetes.io/projected/357ff5a6-97ce-450b-9948-94c31be65cfe-kube-api-access-cccgh\") pod \"kuadrant-operator-controller-manager-86878c769f-957w7\" (UID: \"357ff5a6-97ce-450b-9948-94c31be65cfe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-957w7" Jul 14 18:17:44.275045 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:44.275011 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-957w7" Jul 14 18:17:44.429118 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:44.429073 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-957w7"] Jul 14 18:17:44.431710 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:17:44.431688 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod357ff5a6_97ce_450b_9948_94c31be65cfe.slice/crio-faa9d720508481965e85f0a787aa55a2723cdba12f4ba7c358c295c58942fb18 WatchSource:0}: Error finding container faa9d720508481965e85f0a787aa55a2723cdba12f4ba7c358c295c58942fb18: Status 404 returned error can't find the container with id faa9d720508481965e85f0a787aa55a2723cdba12f4ba7c358c295c58942fb18 Jul 14 18:17:44.653787 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:44.653686 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-957w7" event={"ID":"357ff5a6-97ce-450b-9948-94c31be65cfe","Type":"ContainerStarted","Data":"d09c847b446f128b42c1e9dcf840e6dc8e39800466ba5cbd994f7630ab951eaf"} Jul 14 18:17:44.653787 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:44.653738 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-957w7" event={"ID":"357ff5a6-97ce-450b-9948-94c31be65cfe","Type":"ContainerStarted","Data":"faa9d720508481965e85f0a787aa55a2723cdba12f4ba7c358c295c58942fb18"} Jul 14 18:17:44.653787 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:44.653773 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-957w7" Jul 14 18:17:44.680557 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:44.680511 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-957w7" podStartSLOduration=1.680496442 podStartE2EDuration="1.680496442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:17:44.67846683 +0000 UTC m=+469.515810087" watchObservedRunningTime="2026-07-14 18:17:44.680496442 +0000 UTC m=+469.517839771" Jul 14 18:17:55.659983 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:55.659957 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-957w7" Jul 14 18:17:55.713654 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:55.713618 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rfzgp"] Jul 14 18:17:55.713902 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:55.713862 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rfzgp" podUID="4a1aa1c0-0912-47da-9270-cb270d0cbc24" containerName="manager" containerID="cri-o://f4fbdf5351090de1dd56f45a1b4198f9a62c7c6aa95922ec8fe2ca21d3768a1b" gracePeriod=10 Jul 14 18:17:55.961035 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:55.961009 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rfzgp" Jul 14 18:17:56.054003 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:56.053971 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqr22\" (UniqueName: \"kubernetes.io/projected/4a1aa1c0-0912-47da-9270-cb270d0cbc24-kube-api-access-rqr22\") pod \"4a1aa1c0-0912-47da-9270-cb270d0cbc24\" (UID: \"4a1aa1c0-0912-47da-9270-cb270d0cbc24\") " Jul 14 18:17:56.054207 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:56.054036 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a1aa1c0-0912-47da-9270-cb270d0cbc24-extensions-socket-volume\") pod \"4a1aa1c0-0912-47da-9270-cb270d0cbc24\" (UID: \"4a1aa1c0-0912-47da-9270-cb270d0cbc24\") " Jul 14 18:17:56.054420 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:56.054399 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a1aa1c0-0912-47da-9270-cb270d0cbc24-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4a1aa1c0-0912-47da-9270-cb270d0cbc24" (UID: "4a1aa1c0-0912-47da-9270-cb270d0cbc24"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:17:56.056242 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:56.056218 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a1aa1c0-0912-47da-9270-cb270d0cbc24-kube-api-access-rqr22" (OuterVolumeSpecName: "kube-api-access-rqr22") pod "4a1aa1c0-0912-47da-9270-cb270d0cbc24" (UID: "4a1aa1c0-0912-47da-9270-cb270d0cbc24"). InnerVolumeSpecName "kube-api-access-rqr22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:17:56.155181 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:56.155083 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqr22\" (UniqueName: \"kubernetes.io/projected/4a1aa1c0-0912-47da-9270-cb270d0cbc24-kube-api-access-rqr22\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:17:56.155181 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:56.155124 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a1aa1c0-0912-47da-9270-cb270d0cbc24-extensions-socket-volume\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:17:56.710896 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:56.710861 2565 generic.go:356] "Generic (PLEG): container finished" podID="4a1aa1c0-0912-47da-9270-cb270d0cbc24" containerID="f4fbdf5351090de1dd56f45a1b4198f9a62c7c6aa95922ec8fe2ca21d3768a1b" exitCode=0 Jul 14 18:17:56.711386 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:56.710926 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rfzgp" event={"ID":"4a1aa1c0-0912-47da-9270-cb270d0cbc24","Type":"ContainerDied","Data":"f4fbdf5351090de1dd56f45a1b4198f9a62c7c6aa95922ec8fe2ca21d3768a1b"} Jul 14 18:17:56.711386 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:56.710961 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rfzgp" Jul 14 18:17:56.711386 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:56.710979 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rfzgp" event={"ID":"4a1aa1c0-0912-47da-9270-cb270d0cbc24","Type":"ContainerDied","Data":"51a3900898bbdda4077eece4490c90dd1d711bfa97b1d0dcd5afba673551638a"} Jul 14 18:17:56.711386 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:56.711011 2565 scope.go:117] "RemoveContainer" containerID="f4fbdf5351090de1dd56f45a1b4198f9a62c7c6aa95922ec8fe2ca21d3768a1b" Jul 14 18:17:56.720614 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:56.720590 2565 scope.go:117] "RemoveContainer" containerID="f4fbdf5351090de1dd56f45a1b4198f9a62c7c6aa95922ec8fe2ca21d3768a1b" Jul 14 18:17:56.721018 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:17:56.720987 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4fbdf5351090de1dd56f45a1b4198f9a62c7c6aa95922ec8fe2ca21d3768a1b\": container with ID starting with f4fbdf5351090de1dd56f45a1b4198f9a62c7c6aa95922ec8fe2ca21d3768a1b not found: ID does not exist" containerID="f4fbdf5351090de1dd56f45a1b4198f9a62c7c6aa95922ec8fe2ca21d3768a1b" Jul 14 18:17:56.721168 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:56.721029 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4fbdf5351090de1dd56f45a1b4198f9a62c7c6aa95922ec8fe2ca21d3768a1b"} err="failed to get container status \"f4fbdf5351090de1dd56f45a1b4198f9a62c7c6aa95922ec8fe2ca21d3768a1b\": rpc error: code = NotFound desc = could not find container \"f4fbdf5351090de1dd56f45a1b4198f9a62c7c6aa95922ec8fe2ca21d3768a1b\": container with ID starting with f4fbdf5351090de1dd56f45a1b4198f9a62c7c6aa95922ec8fe2ca21d3768a1b not found: ID does not exist" Jul 14 18:17:56.741467 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:56.741440 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rfzgp"] Jul 14 18:17:56.753287 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:56.753260 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rfzgp"] Jul 14 18:17:57.689540 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:17:57.689511 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a1aa1c0-0912-47da-9270-cb270d0cbc24" path="/var/lib/kubelet/pods/4a1aa1c0-0912-47da-9270-cb270d0cbc24/volumes" Jul 14 18:18:00.097718 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.097686 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr"] Jul 14 18:18:00.098141 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.098124 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a1aa1c0-0912-47da-9270-cb270d0cbc24" containerName="manager" Jul 14 18:18:00.098218 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.098146 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a1aa1c0-0912-47da-9270-cb270d0cbc24" containerName="manager" Jul 14 18:18:00.098272 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.098245 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a1aa1c0-0912-47da-9270-cb270d0cbc24" containerName="manager" Jul 14 18:18:00.102716 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.102697 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:00.105343 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.105318 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-px64t\"" Jul 14 18:18:00.114335 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.114308 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr"] Jul 14 18:18:00.188472 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.188442 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac2ef1e-824e-482f-8087-dc90fa894ebd-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sv6fr\" (UID: \"6ac2ef1e-824e-482f-8087-dc90fa894ebd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:00.188657 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.188491 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac2ef1e-824e-482f-8087-dc90fa894ebd-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sv6fr\" (UID: \"6ac2ef1e-824e-482f-8087-dc90fa894ebd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:00.188657 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.188518 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac2ef1e-824e-482f-8087-dc90fa894ebd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sv6fr\" (UID: \"6ac2ef1e-824e-482f-8087-dc90fa894ebd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:00.188657 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.188596 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ac2ef1e-824e-482f-8087-dc90fa894ebd-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sv6fr\" (UID: \"6ac2ef1e-824e-482f-8087-dc90fa894ebd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:00.188657 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.188633 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac2ef1e-824e-482f-8087-dc90fa894ebd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sv6fr\" (UID: \"6ac2ef1e-824e-482f-8087-dc90fa894ebd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:00.188800 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.188675 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac2ef1e-824e-482f-8087-dc90fa894ebd-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sv6fr\" (UID: \"6ac2ef1e-824e-482f-8087-dc90fa894ebd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:00.188800 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.188696 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac2ef1e-824e-482f-8087-dc90fa894ebd-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sv6fr\" (UID: \"6ac2ef1e-824e-482f-8087-dc90fa894ebd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:00.188800 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.188718 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmt5w\" (UniqueName: \"kubernetes.io/projected/6ac2ef1e-824e-482f-8087-dc90fa894ebd-kube-api-access-rmt5w\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sv6fr\" (UID: \"6ac2ef1e-824e-482f-8087-dc90fa894ebd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:00.188800 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.188792 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac2ef1e-824e-482f-8087-dc90fa894ebd-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sv6fr\" (UID: \"6ac2ef1e-824e-482f-8087-dc90fa894ebd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:00.289346 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.289315 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac2ef1e-824e-482f-8087-dc90fa894ebd-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sv6fr\" (UID: \"6ac2ef1e-824e-482f-8087-dc90fa894ebd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:00.289346 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.289353 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac2ef1e-824e-482f-8087-dc90fa894ebd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sv6fr\" (UID: \"6ac2ef1e-824e-482f-8087-dc90fa894ebd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:00.289621 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.289392 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ac2ef1e-824e-482f-8087-dc90fa894ebd-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sv6fr\" (UID: \"6ac2ef1e-824e-482f-8087-dc90fa894ebd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:00.289621 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.289417 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac2ef1e-824e-482f-8087-dc90fa894ebd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sv6fr\" (UID: \"6ac2ef1e-824e-482f-8087-dc90fa894ebd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:00.289621 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.289457 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac2ef1e-824e-482f-8087-dc90fa894ebd-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sv6fr\" (UID: \"6ac2ef1e-824e-482f-8087-dc90fa894ebd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:00.289621 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.289482 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac2ef1e-824e-482f-8087-dc90fa894ebd-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sv6fr\" (UID: \"6ac2ef1e-824e-482f-8087-dc90fa894ebd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:00.289621 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.289511 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmt5w\" (UniqueName: \"kubernetes.io/projected/6ac2ef1e-824e-482f-8087-dc90fa894ebd-kube-api-access-rmt5w\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sv6fr\" (UID: \"6ac2ef1e-824e-482f-8087-dc90fa894ebd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:00.289621 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.289571 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac2ef1e-824e-482f-8087-dc90fa894ebd-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sv6fr\" (UID: \"6ac2ef1e-824e-482f-8087-dc90fa894ebd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:00.289621 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.289615 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac2ef1e-824e-482f-8087-dc90fa894ebd-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sv6fr\" (UID: \"6ac2ef1e-824e-482f-8087-dc90fa894ebd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:00.290020 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.289908 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac2ef1e-824e-482f-8087-dc90fa894ebd-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sv6fr\" (UID: \"6ac2ef1e-824e-482f-8087-dc90fa894ebd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:00.290020 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.289920 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac2ef1e-824e-482f-8087-dc90fa894ebd-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sv6fr\" (UID: \"6ac2ef1e-824e-482f-8087-dc90fa894ebd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:00.290225 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.290206 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac2ef1e-824e-482f-8087-dc90fa894ebd-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sv6fr\" (UID: \"6ac2ef1e-824e-482f-8087-dc90fa894ebd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:00.290225 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.290216 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac2ef1e-824e-482f-8087-dc90fa894ebd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sv6fr\" (UID: \"6ac2ef1e-824e-482f-8087-dc90fa894ebd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:00.290321 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.290258 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac2ef1e-824e-482f-8087-dc90fa894ebd-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sv6fr\" (UID: \"6ac2ef1e-824e-482f-8087-dc90fa894ebd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:00.292325 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.292304 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac2ef1e-824e-482f-8087-dc90fa894ebd-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sv6fr\" (UID: \"6ac2ef1e-824e-482f-8087-dc90fa894ebd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:00.292637 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.292614 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac2ef1e-824e-482f-8087-dc90fa894ebd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sv6fr\" (UID: \"6ac2ef1e-824e-482f-8087-dc90fa894ebd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:00.299480 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.299455 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmt5w\" (UniqueName: \"kubernetes.io/projected/6ac2ef1e-824e-482f-8087-dc90fa894ebd-kube-api-access-rmt5w\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sv6fr\" (UID: \"6ac2ef1e-824e-482f-8087-dc90fa894ebd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:00.299595 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.299458 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ac2ef1e-824e-482f-8087-dc90fa894ebd-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-sv6fr\" (UID: \"6ac2ef1e-824e-482f-8087-dc90fa894ebd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:00.413370 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.413275 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:00.555739 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.555713 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr"] Jul 14 18:18:00.557747 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:18:00.557717 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac2ef1e_824e_482f_8087_dc90fa894ebd.slice/crio-eee57a558a3283ff50c4e4eefbf7cbc2cc296422c37e202a95b8db607c41e64b WatchSource:0}: Error finding container eee57a558a3283ff50c4e4eefbf7cbc2cc296422c37e202a95b8db607c41e64b: Status 404 returned error can't find the container with id eee57a558a3283ff50c4e4eefbf7cbc2cc296422c37e202a95b8db607c41e64b Jul 14 18:18:00.731954 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:00.731861 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" event={"ID":"6ac2ef1e-824e-482f-8087-dc90fa894ebd","Type":"ContainerStarted","Data":"eee57a558a3283ff50c4e4eefbf7cbc2cc296422c37e202a95b8db607c41e64b"} Jul 14 18:18:03.313059 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:03.313028 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:18:03.313436 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:03.313127 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:18:03.313436 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:03.313181 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:18:03.748140 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:03.748021 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" event={"ID":"6ac2ef1e-824e-482f-8087-dc90fa894ebd","Type":"ContainerStarted","Data":"d0b7ce94288ead6e19db9af6998cd0e022dd6f0568fb252ffc53818f6c3d1b37"} Jul 14 18:18:03.776019 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:03.775941 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" podStartSLOduration=1.023245379 podStartE2EDuration="3.775922945s" totalImagesPullingTime="2.752677566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:18:03.775214437 +0000 UTC m=+488.612557695" watchObservedRunningTime="2026-07-14 18:18:03.775922945 +0000 UTC m=+488.613266203" Jul 14 18:18:04.414411 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:04.414371 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:04.420007 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:04.419983 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:04.752417 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:04.752327 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:04.753378 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:04.753360 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:18:13.829634 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:13.829603 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:18:13.833571 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:13.833545 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-ztrvp" Jul 14 18:18:13.836650 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:13.836617 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 14 18:18:13.836812 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:13.836697 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-n85pf\"" Jul 14 18:18:13.846325 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:13.846297 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:18:13.890908 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:13.890873 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:18:13.906641 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:13.906610 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf4fm\" (UniqueName: \"kubernetes.io/projected/bf5effef-70b1-40d2-9959-537f4ca0f0a1-kube-api-access-tf4fm\") pod \"limitador-limitador-54d6d988d5-ztrvp\" (UID: \"bf5effef-70b1-40d2-9959-537f4ca0f0a1\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ztrvp" Jul 14 18:18:13.906796 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:13.906669 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bf5effef-70b1-40d2-9959-537f4ca0f0a1-config-file\") pod \"limitador-limitador-54d6d988d5-ztrvp\" (UID: \"bf5effef-70b1-40d2-9959-537f4ca0f0a1\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ztrvp" Jul 14 18:18:14.008163 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:14.008094 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf4fm\" (UniqueName: \"kubernetes.io/projected/bf5effef-70b1-40d2-9959-537f4ca0f0a1-kube-api-access-tf4fm\") pod \"limitador-limitador-54d6d988d5-ztrvp\" (UID: \"bf5effef-70b1-40d2-9959-537f4ca0f0a1\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ztrvp" Jul 14 18:18:14.008359 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:14.008179 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bf5effef-70b1-40d2-9959-537f4ca0f0a1-config-file\") pod \"limitador-limitador-54d6d988d5-ztrvp\" (UID: \"bf5effef-70b1-40d2-9959-537f4ca0f0a1\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ztrvp" Jul 14 18:18:14.008837 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:14.008811 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bf5effef-70b1-40d2-9959-537f4ca0f0a1-config-file\") pod \"limitador-limitador-54d6d988d5-ztrvp\" (UID: \"bf5effef-70b1-40d2-9959-537f4ca0f0a1\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ztrvp" Jul 14 18:18:14.018696 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:14.018665 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf4fm\" (UniqueName: \"kubernetes.io/projected/bf5effef-70b1-40d2-9959-537f4ca0f0a1-kube-api-access-tf4fm\") pod \"limitador-limitador-54d6d988d5-ztrvp\" (UID: \"bf5effef-70b1-40d2-9959-537f4ca0f0a1\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ztrvp" Jul 14 18:18:14.146371 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:14.146279 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-ztrvp" Jul 14 18:18:14.315194 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:14.315164 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:18:14.322378 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:18:14.322338 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf5effef_70b1_40d2_9959_537f4ca0f0a1.slice/crio-6e69ca5c2307f4878ddd5e80d0403518d08210c039269d4afb450807c852ac6d WatchSource:0}: Error finding container 6e69ca5c2307f4878ddd5e80d0403518d08210c039269d4afb450807c852ac6d: Status 404 returned error can't find the container with id 6e69ca5c2307f4878ddd5e80d0403518d08210c039269d4afb450807c852ac6d Jul 14 18:18:14.801779 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:14.801718 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-ztrvp" event={"ID":"bf5effef-70b1-40d2-9959-537f4ca0f0a1","Type":"ContainerStarted","Data":"6e69ca5c2307f4878ddd5e80d0403518d08210c039269d4afb450807c852ac6d"} Jul 14 18:18:17.818732 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:17.818690 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-ztrvp" event={"ID":"bf5effef-70b1-40d2-9959-537f4ca0f0a1","Type":"ContainerStarted","Data":"6008b38911d3c9b251ef2ce7843d6a4abca7a8855a90d05969651b0e4a0543e5"} Jul 14 18:18:17.819138 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:17.818765 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-ztrvp" Jul 14 18:18:17.839259 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:17.839216 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-ztrvp" podStartSLOduration=2.322057486 podStartE2EDuration="4.839200174s" totalImagesPullingTime="2.517142688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:18:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:18:17.838344672 +0000 UTC m=+502.675687930" watchObservedRunningTime="2026-07-14 18:18:17.839200174 +0000 UTC m=+502.676543462" Jul 14 18:18:28.824649 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:18:28.824618 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-ztrvp" Jul 14 18:19:28.962670 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:19:28.962632 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:19:35.794297 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:19:35.794253 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-sv6fr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 14 18:19:35.794785 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:19:35.794328 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" podUID="6ac2ef1e-824e-482f-8087-dc90fa894ebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 14 18:19:37.171027 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:19:37.170989 2565 generic.go:356] "Generic (PLEG): container finished" podID="6ac2ef1e-824e-482f-8087-dc90fa894ebd" containerID="d0b7ce94288ead6e19db9af6998cd0e022dd6f0568fb252ffc53818f6c3d1b37" exitCode=0 Jul 14 18:19:37.171466 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:19:37.171061 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" event={"ID":"6ac2ef1e-824e-482f-8087-dc90fa894ebd","Type":"ContainerDied","Data":"d0b7ce94288ead6e19db9af6998cd0e022dd6f0568fb252ffc53818f6c3d1b37"} Jul 14 18:19:37.171514 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:19:37.171470 2565 scope.go:117] "RemoveContainer" containerID="d0b7ce94288ead6e19db9af6998cd0e022dd6f0568fb252ffc53818f6c3d1b37" Jul 14 18:19:37.172469 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:19:37.172443 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:19:37.172538 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:19:37.172525 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:19:37.172576 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:19:37.172559 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:19:38.176653 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:19:38.176617 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" event={"ID":"6ac2ef1e-824e-482f-8087-dc90fa894ebd","Type":"ContainerStarted","Data":"12b17bdac80240f7c28c6b634ea87ba7be6235fc3574079c886625692f61ffb7"} Jul 14 18:19:38.413479 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:19:38.413422 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:19:38.413898 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:19:38.413873 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-sv6fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Jul 14 18:19:38.413965 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:19:38.413944 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" podUID="6ac2ef1e-824e-482f-8087-dc90fa894ebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Jul 14 18:19:39.413790 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:19:39.413753 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-sv6fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Jul 14 18:19:39.414215 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:19:39.413814 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" podUID="6ac2ef1e-824e-482f-8087-dc90fa894ebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Jul 14 18:19:40.414347 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:19:40.414308 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-sv6fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Jul 14 18:19:40.414833 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:19:40.414598 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" podUID="6ac2ef1e-824e-482f-8087-dc90fa894ebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Jul 14 18:19:41.414564 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:19:41.414517 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-sv6fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Jul 14 18:19:41.415081 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:19:41.414593 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" podUID="6ac2ef1e-824e-482f-8087-dc90fa894ebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Jul 14 18:19:42.417211 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:19:42.417179 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:19:42.417596 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:19:42.417464 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:19:42.418074 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:19:42.418058 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-sv6fr" Jul 14 18:19:47.664208 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:19:47.664171 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:19:55.605729 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:19:55.605701 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wqpt2_4525dcda-ead6-4800-8d17-58812c289010/console-operator/1.log" Jul 14 18:19:55.608789 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:19:55.608765 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f99bz_5f0ecb09-5c7f-4709-9b4a-1a6a03580822/ovn-acl-logging/0.log" Jul 14 18:19:55.609067 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:19:55.609049 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wqpt2_4525dcda-ead6-4800-8d17-58812c289010/console-operator/1.log" Jul 14 18:19:55.612499 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:19:55.612481 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f99bz_5f0ecb09-5c7f-4709-9b4a-1a6a03580822/ovn-acl-logging/0.log" Jul 14 18:19:57.961906 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:19:57.961872 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:20:04.963777 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:04.963742 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-lpdjq"] Jul 14 18:20:04.966716 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:04.966698 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lpdjq" Jul 14 18:20:04.971307 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:04.971283 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 14 18:20:04.972529 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:04.972509 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6s2bs\"" Jul 14 18:20:04.972684 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:04.972527 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 14 18:20:04.972772 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:04.972630 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 14 18:20:04.987213 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:04.987185 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-lpdjq"] Jul 14 18:20:05.058330 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:05.058292 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b21e27-b560-4e7e-a2c8-7455b0017a30-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lpdjq\" (UID: \"76b21e27-b560-4e7e-a2c8-7455b0017a30\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lpdjq" Jul 14 18:20:05.058525 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:05.058331 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drgmx\" (UniqueName: \"kubernetes.io/projected/76b21e27-b560-4e7e-a2c8-7455b0017a30-kube-api-access-drgmx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lpdjq\" (UID: \"76b21e27-b560-4e7e-a2c8-7455b0017a30\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lpdjq" Jul 14 18:20:05.058525 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:05.058463 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b21e27-b560-4e7e-a2c8-7455b0017a30-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lpdjq\" (UID: \"76b21e27-b560-4e7e-a2c8-7455b0017a30\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lpdjq" Jul 14 18:20:05.058661 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:05.058522 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b21e27-b560-4e7e-a2c8-7455b0017a30-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lpdjq\" (UID: \"76b21e27-b560-4e7e-a2c8-7455b0017a30\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lpdjq" Jul 14 18:20:05.058661 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:05.058574 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b21e27-b560-4e7e-a2c8-7455b0017a30-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lpdjq\" (UID: \"76b21e27-b560-4e7e-a2c8-7455b0017a30\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lpdjq" Jul 14 18:20:05.058661 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:05.058632 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b21e27-b560-4e7e-a2c8-7455b0017a30-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lpdjq\" (UID: \"76b21e27-b560-4e7e-a2c8-7455b0017a30\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lpdjq" Jul 14 18:20:05.159588 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:05.159544 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b21e27-b560-4e7e-a2c8-7455b0017a30-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lpdjq\" (UID: \"76b21e27-b560-4e7e-a2c8-7455b0017a30\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lpdjq" Jul 14 18:20:05.159797 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:05.159593 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drgmx\" (UniqueName: \"kubernetes.io/projected/76b21e27-b560-4e7e-a2c8-7455b0017a30-kube-api-access-drgmx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lpdjq\" (UID: \"76b21e27-b560-4e7e-a2c8-7455b0017a30\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lpdjq" Jul 14 18:20:05.159797 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:05.159643 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b21e27-b560-4e7e-a2c8-7455b0017a30-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lpdjq\" (UID: \"76b21e27-b560-4e7e-a2c8-7455b0017a30\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lpdjq" Jul 14 18:20:05.159797 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:05.159682 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b21e27-b560-4e7e-a2c8-7455b0017a30-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lpdjq\" (UID: \"76b21e27-b560-4e7e-a2c8-7455b0017a30\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lpdjq" Jul 14 18:20:05.159797 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:05.159725 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b21e27-b560-4e7e-a2c8-7455b0017a30-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lpdjq\" (UID: \"76b21e27-b560-4e7e-a2c8-7455b0017a30\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lpdjq" Jul 14 18:20:05.159797 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:05.159772 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b21e27-b560-4e7e-a2c8-7455b0017a30-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lpdjq\" (UID: \"76b21e27-b560-4e7e-a2c8-7455b0017a30\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lpdjq" Jul 14 18:20:05.160132 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:05.159957 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b21e27-b560-4e7e-a2c8-7455b0017a30-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lpdjq\" (UID: \"76b21e27-b560-4e7e-a2c8-7455b0017a30\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lpdjq" Jul 14 18:20:05.160203 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:05.160134 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b21e27-b560-4e7e-a2c8-7455b0017a30-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lpdjq\" (UID: \"76b21e27-b560-4e7e-a2c8-7455b0017a30\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lpdjq" Jul 14 18:20:05.160203 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:05.160184 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b21e27-b560-4e7e-a2c8-7455b0017a30-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lpdjq\" (UID: \"76b21e27-b560-4e7e-a2c8-7455b0017a30\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lpdjq" Jul 14 18:20:05.162223 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:05.162193 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b21e27-b560-4e7e-a2c8-7455b0017a30-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lpdjq\" (UID: \"76b21e27-b560-4e7e-a2c8-7455b0017a30\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lpdjq" Jul 14 18:20:05.162512 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:05.162491 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b21e27-b560-4e7e-a2c8-7455b0017a30-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lpdjq\" (UID: \"76b21e27-b560-4e7e-a2c8-7455b0017a30\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lpdjq" Jul 14 18:20:05.171249 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:05.171217 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drgmx\" (UniqueName: \"kubernetes.io/projected/76b21e27-b560-4e7e-a2c8-7455b0017a30-kube-api-access-drgmx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lpdjq\" (UID: \"76b21e27-b560-4e7e-a2c8-7455b0017a30\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lpdjq" Jul 14 18:20:05.278024 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:05.277979 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lpdjq" Jul 14 18:20:05.423351 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:05.423325 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-lpdjq"] Jul 14 18:20:05.425401 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:20:05.425375 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b21e27_b560_4e7e_a2c8_7455b0017a30.slice/crio-664f2322c9c4ad34c419a8bd8aadc0f738cc38b1455165603eae592967c9a3a2 WatchSource:0}: Error finding container 664f2322c9c4ad34c419a8bd8aadc0f738cc38b1455165603eae592967c9a3a2: Status 404 returned error can't find the container with id 664f2322c9c4ad34c419a8bd8aadc0f738cc38b1455165603eae592967c9a3a2 Jul 14 18:20:06.176033 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:06.175999 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:20:06.304844 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:06.304809 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lpdjq" event={"ID":"76b21e27-b560-4e7e-a2c8-7455b0017a30","Type":"ContainerStarted","Data":"664f2322c9c4ad34c419a8bd8aadc0f738cc38b1455165603eae592967c9a3a2"} Jul 14 18:20:11.328605 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:11.328566 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lpdjq" event={"ID":"76b21e27-b560-4e7e-a2c8-7455b0017a30","Type":"ContainerStarted","Data":"4a4e80e7dc813ae8f3b4b4a277745995fae400b36ef7978ab64fd61ac298dba6"} Jul 14 18:20:11.354597 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:11.354543 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lpdjq" podStartSLOduration=2.469989465 podStartE2EDuration="7.354526784s" totalImagesPullingTime="4.884537319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:20:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:20:11.353744362 +0000 UTC m=+616.191087620" watchObservedRunningTime="2026-07-14 18:20:11.354526784 +0000 UTC m=+616.191870041" Jul 14 18:20:12.484734 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:12.484700 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:20:15.278404 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:15.278357 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lpdjq" Jul 14 18:20:15.278849 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:15.278460 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lpdjq" Jul 14 18:20:15.291092 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:15.291058 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lpdjq" Jul 14 18:20:15.359350 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:15.359318 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lpdjq" Jul 14 18:20:23.960519 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:23.960427 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:20:33.255289 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:20:33.255254 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:22:09.285318 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:22:09.285276 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:22:22.370980 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:22:22.370943 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:22:31.672306 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:22:31.672268 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:22:41.857868 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:22:41.857833 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:22:51.255808 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:22:51.255777 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:23:18.761720 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:23:18.761626 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:23:30.464549 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:23:30.464516 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:24:04.754901 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:24:04.754862 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:24:13.758906 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:24:13.758868 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:24:23.763508 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:24:23.763473 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:24:31.448697 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:24:31.448662 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:24:40.956579 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:24:40.956547 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:24:51.969987 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:24:51.969903 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:24:55.633083 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:24:55.633053 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wqpt2_4525dcda-ead6-4800-8d17-58812c289010/console-operator/1.log" Jul 14 18:24:55.634606 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:24:55.634584 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wqpt2_4525dcda-ead6-4800-8d17-58812c289010/console-operator/1.log" Jul 14 18:24:55.636347 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:24:55.636330 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f99bz_5f0ecb09-5c7f-4709-9b4a-1a6a03580822/ovn-acl-logging/0.log" Jul 14 18:24:55.637818 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:24:55.637800 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f99bz_5f0ecb09-5c7f-4709-9b4a-1a6a03580822/ovn-acl-logging/0.log" Jul 14 18:25:02.471605 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:25:02.471571 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:25:11.952486 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:25:11.952452 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:25:37.652407 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:25:37.652363 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:25:47.359207 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:25:47.359169 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:26:11.861366 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:26:11.861333 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:26:24.563896 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:26:24.563814 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:26:40.153262 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:26:40.153226 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:26:48.964295 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:26:48.964253 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:26:58.377449 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:26:58.377407 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:27:07.752342 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:27:07.752309 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:27:17.851912 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:27:17.851877 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:27:44.079263 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:27:44.079229 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-957w7"] Jul 14 18:27:44.079694 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:27:44.079460 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-957w7" podUID="357ff5a6-97ce-450b-9948-94c31be65cfe" containerName="manager" containerID="cri-o://d09c847b446f128b42c1e9dcf840e6dc8e39800466ba5cbd994f7630ab951eaf" gracePeriod=10 Jul 14 18:27:47.236146 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:27:47.236119 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-957w7" Jul 14 18:27:47.330157 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:27:47.330129 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cccgh\" (UniqueName: \"kubernetes.io/projected/357ff5a6-97ce-450b-9948-94c31be65cfe-kube-api-access-cccgh\") pod \"357ff5a6-97ce-450b-9948-94c31be65cfe\" (UID: \"357ff5a6-97ce-450b-9948-94c31be65cfe\") " Jul 14 18:27:47.330345 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:27:47.330214 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/357ff5a6-97ce-450b-9948-94c31be65cfe-extensions-socket-volume\") pod \"357ff5a6-97ce-450b-9948-94c31be65cfe\" (UID: \"357ff5a6-97ce-450b-9948-94c31be65cfe\") " Jul 14 18:27:47.330564 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:27:47.330541 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357ff5a6-97ce-450b-9948-94c31be65cfe-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "357ff5a6-97ce-450b-9948-94c31be65cfe" (UID: "357ff5a6-97ce-450b-9948-94c31be65cfe"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:27:47.332283 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:27:47.332263 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/357ff5a6-97ce-450b-9948-94c31be65cfe-kube-api-access-cccgh" (OuterVolumeSpecName: "kube-api-access-cccgh") pod "357ff5a6-97ce-450b-9948-94c31be65cfe" (UID: "357ff5a6-97ce-450b-9948-94c31be65cfe"). InnerVolumeSpecName "kube-api-access-cccgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:27:47.411521 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:27:47.411495 2565 generic.go:356] "Generic (PLEG): container finished" podID="357ff5a6-97ce-450b-9948-94c31be65cfe" containerID="d09c847b446f128b42c1e9dcf840e6dc8e39800466ba5cbd994f7630ab951eaf" exitCode=0 Jul 14 18:27:47.411678 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:27:47.411528 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-957w7" event={"ID":"357ff5a6-97ce-450b-9948-94c31be65cfe","Type":"ContainerDied","Data":"d09c847b446f128b42c1e9dcf840e6dc8e39800466ba5cbd994f7630ab951eaf"} Jul 14 18:27:47.411678 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:27:47.411561 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-957w7" Jul 14 18:27:47.411678 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:27:47.411573 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-957w7" event={"ID":"357ff5a6-97ce-450b-9948-94c31be65cfe","Type":"ContainerDied","Data":"faa9d720508481965e85f0a787aa55a2723cdba12f4ba7c358c295c58942fb18"} Jul 14 18:27:47.411678 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:27:47.411594 2565 scope.go:117] "RemoveContainer" containerID="d09c847b446f128b42c1e9dcf840e6dc8e39800466ba5cbd994f7630ab951eaf" Jul 14 18:27:47.423327 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:27:47.423309 2565 scope.go:117] "RemoveContainer" containerID="d09c847b446f128b42c1e9dcf840e6dc8e39800466ba5cbd994f7630ab951eaf" Jul 14 18:27:47.423551 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:27:47.423533 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d09c847b446f128b42c1e9dcf840e6dc8e39800466ba5cbd994f7630ab951eaf\": container with ID starting with d09c847b446f128b42c1e9dcf840e6dc8e39800466ba5cbd994f7630ab951eaf not found: ID does not exist" containerID="d09c847b446f128b42c1e9dcf840e6dc8e39800466ba5cbd994f7630ab951eaf" Jul 14 18:27:47.423609 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:27:47.423561 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d09c847b446f128b42c1e9dcf840e6dc8e39800466ba5cbd994f7630ab951eaf"} err="failed to get container status \"d09c847b446f128b42c1e9dcf840e6dc8e39800466ba5cbd994f7630ab951eaf\": rpc error: code = NotFound desc = could not find container \"d09c847b446f128b42c1e9dcf840e6dc8e39800466ba5cbd994f7630ab951eaf\": container with ID starting with d09c847b446f128b42c1e9dcf840e6dc8e39800466ba5cbd994f7630ab951eaf not found: ID does not exist" Jul 14 18:27:47.431614 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:27:47.431595 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/357ff5a6-97ce-450b-9948-94c31be65cfe-extensions-socket-volume\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:27:47.431614 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:27:47.431616 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cccgh\" (UniqueName: \"kubernetes.io/projected/357ff5a6-97ce-450b-9948-94c31be65cfe-kube-api-access-cccgh\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:27:47.437006 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:27:47.436985 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-957w7"] Jul 14 18:27:47.445980 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:27:47.445960 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-957w7"] Jul 14 18:27:47.689532 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:27:47.689458 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="357ff5a6-97ce-450b-9948-94c31be65cfe" path="/var/lib/kubelet/pods/357ff5a6-97ce-450b-9948-94c31be65cfe/volumes" Jul 14 18:28:45.279125 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:28:45.279061 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bzrwb"] Jul 14 18:28:45.279611 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:28:45.279393 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="357ff5a6-97ce-450b-9948-94c31be65cfe" containerName="manager" Jul 14 18:28:45.279611 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:28:45.279404 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="357ff5a6-97ce-450b-9948-94c31be65cfe" containerName="manager" Jul 14 18:28:45.279611 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:28:45.279465 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="357ff5a6-97ce-450b-9948-94c31be65cfe" containerName="manager" Jul 14 18:28:45.282373 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:28:45.282357 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bzrwb" Jul 14 18:28:45.286206 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:28:45.286183 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-k5cr2\"" Jul 14 18:28:45.303402 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:28:45.303379 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bzrwb"] Jul 14 18:28:45.310921 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:28:45.310900 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgwgj\" (UniqueName: \"kubernetes.io/projected/cebd3785-87d9-41f2-883c-f96424b89823-kube-api-access-xgwgj\") pod \"kuadrant-operator-controller-manager-86878c769f-bzrwb\" (UID: \"cebd3785-87d9-41f2-883c-f96424b89823\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bzrwb" Jul 14 18:28:45.311016 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:28:45.310936 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cebd3785-87d9-41f2-883c-f96424b89823-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bzrwb\" (UID: \"cebd3785-87d9-41f2-883c-f96424b89823\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bzrwb" Jul 14 18:28:45.412176 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:28:45.412147 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgwgj\" (UniqueName: \"kubernetes.io/projected/cebd3785-87d9-41f2-883c-f96424b89823-kube-api-access-xgwgj\") pod \"kuadrant-operator-controller-manager-86878c769f-bzrwb\" (UID: \"cebd3785-87d9-41f2-883c-f96424b89823\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bzrwb" Jul 14 18:28:45.412308 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:28:45.412202 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cebd3785-87d9-41f2-883c-f96424b89823-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bzrwb\" (UID: \"cebd3785-87d9-41f2-883c-f96424b89823\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bzrwb" Jul 14 18:28:45.412624 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:28:45.412607 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cebd3785-87d9-41f2-883c-f96424b89823-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bzrwb\" (UID: \"cebd3785-87d9-41f2-883c-f96424b89823\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bzrwb" Jul 14 18:28:45.423214 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:28:45.423186 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgwgj\" (UniqueName: \"kubernetes.io/projected/cebd3785-87d9-41f2-883c-f96424b89823-kube-api-access-xgwgj\") pod \"kuadrant-operator-controller-manager-86878c769f-bzrwb\" (UID: \"cebd3785-87d9-41f2-883c-f96424b89823\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bzrwb" Jul 14 18:28:45.592615 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:28:45.592533 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bzrwb" Jul 14 18:28:45.753483 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:28:45.753455 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bzrwb"] Jul 14 18:28:45.756352 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:28:45.756325 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcebd3785_87d9_41f2_883c_f96424b89823.slice/crio-fc12b7c710ddfdf8f7d9653e9c79317512878664f1cec61a4e0171efef6db22b WatchSource:0}: Error finding container fc12b7c710ddfdf8f7d9653e9c79317512878664f1cec61a4e0171efef6db22b: Status 404 returned error can't find the container with id fc12b7c710ddfdf8f7d9653e9c79317512878664f1cec61a4e0171efef6db22b Jul 14 18:28:46.672025 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:28:46.671991 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bzrwb" event={"ID":"cebd3785-87d9-41f2-883c-f96424b89823","Type":"ContainerStarted","Data":"c9968983134487d1fc9399dc8ca017f1b43ce305da2c13b6e55674247eef9b85"} Jul 14 18:28:46.672442 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:28:46.672039 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bzrwb" event={"ID":"cebd3785-87d9-41f2-883c-f96424b89823","Type":"ContainerStarted","Data":"fc12b7c710ddfdf8f7d9653e9c79317512878664f1cec61a4e0171efef6db22b"} Jul 14 18:28:46.672442 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:28:46.672082 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bzrwb" Jul 14 18:28:46.712188 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:28:46.712138 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bzrwb" podStartSLOduration=1.712116595 podStartE2EDuration="1.712116595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:28:46.710834467 +0000 UTC m=+1131.548177724" watchObservedRunningTime="2026-07-14 18:28:46.712116595 +0000 UTC m=+1131.549459851" Jul 14 18:28:57.679626 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:28:57.679598 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bzrwb" Jul 14 18:29:05.661816 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:29:05.661781 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:29:17.360923 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:29:17.360832 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:29:25.558967 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:29:25.558932 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:29:36.661554 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:29:36.661515 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:29:46.654460 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:29:46.654425 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:29:55.659776 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:29:55.659750 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wqpt2_4525dcda-ead6-4800-8d17-58812c289010/console-operator/1.log" Jul 14 18:29:55.660452 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:29:55.660435 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wqpt2_4525dcda-ead6-4800-8d17-58812c289010/console-operator/1.log" Jul 14 18:29:55.662679 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:29:55.662655 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f99bz_5f0ecb09-5c7f-4709-9b4a-1a6a03580822/ovn-acl-logging/0.log" Jul 14 18:29:55.663499 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:29:55.663480 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f99bz_5f0ecb09-5c7f-4709-9b4a-1a6a03580822/ovn-acl-logging/0.log" Jul 14 18:29:57.250492 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:29:57.250459 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:30:07.353959 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:30:07.353924 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:30:18.256154 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:30:18.256122 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:30:28.959923 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:30:28.959890 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:30:38.553671 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:30:38.553641 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:30:48.355563 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:30:48.355482 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:31:13.054450 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:31:13.054418 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:31:20.159861 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:31:20.159832 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:31:29.056592 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:31:29.056553 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:31:41.461525 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:31:41.461493 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:32:00.051246 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:32:00.051210 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:32:09.361539 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:32:09.361505 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:32:17.977460 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:32:17.977381 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:32:27.350157 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:32:27.350119 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:32:37.651817 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:32:37.651782 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:32:46.844486 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:32:46.844452 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:33:01.153867 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:33:01.153831 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:33:10.150230 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:33:10.150198 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:33:19.670413 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:33:19.670381 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:33:31.152143 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:33:31.152089 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:33:42.369279 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:33:42.369244 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:33:52.856194 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:33:52.856088 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:34:06.163218 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:34:06.163177 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:34:18.955187 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:34:18.955148 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:34:30.857596 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:34:30.857548 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:34:45.862842 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:34:45.862804 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:34:55.695089 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:34:55.695058 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wqpt2_4525dcda-ead6-4800-8d17-58812c289010/console-operator/1.log" Jul 14 18:34:55.697613 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:34:55.695213 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wqpt2_4525dcda-ead6-4800-8d17-58812c289010/console-operator/1.log" Jul 14 18:34:55.698357 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:34:55.698336 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f99bz_5f0ecb09-5c7f-4709-9b4a-1a6a03580822/ovn-acl-logging/0.log" Jul 14 18:34:55.698830 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:34:55.698810 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f99bz_5f0ecb09-5c7f-4709-9b4a-1a6a03580822/ovn-acl-logging/0.log" Jul 14 18:34:57.151972 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:34:57.151937 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:35:06.953782 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:06.953746 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:35:07.911377 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:07.911341 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr"] Jul 14 18:35:07.915705 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:07.915679 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:07.919358 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:07.919329 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f37aedf1-openshift-default-dockercfg-ndgwh\"" Jul 14 18:35:07.936399 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:07.936351 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr"] Jul 14 18:35:07.985743 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:07.985687 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-credential-socket\") pod \"e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:07.985743 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:07.985738 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-workload-socket\") pod \"e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:07.986313 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:07.985794 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqskj\" (UniqueName: \"kubernetes.io/projected/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-kube-api-access-vqskj\") pod \"e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:07.986313 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:07.985841 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-workload-certs\") pod \"e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:07.986313 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:07.985892 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-istiod-ca-cert\") pod \"e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:07.986313 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:07.985948 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-istio-token\") pod \"e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:07.986313 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:07.985999 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-istio-data\") pod \"e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:07.986313 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:07.986026 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-istio-podinfo\") pod \"e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:07.986313 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:07.986157 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-istio-envoy\") pod \"e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:08.086995 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:08.086955 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-credential-socket\") pod \"e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:08.086995 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:08.086995 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-workload-socket\") pod \"e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:08.087270 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:08.087022 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqskj\" (UniqueName: \"kubernetes.io/projected/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-kube-api-access-vqskj\") pod \"e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:08.087270 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:08.087054 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-workload-certs\") pod \"e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:08.087439 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:08.087417 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-istiod-ca-cert\") pod \"e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:08.087993 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:08.087434 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-workload-socket\") pod \"e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:08.087993 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:08.087678 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-istio-token\") pod \"e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:08.087993 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:08.087785 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-workload-certs\") pod \"e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:08.087993 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:08.087800 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-istio-data\") pod \"e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:08.087993 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:08.087827 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-credential-socket\") pod \"e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:08.087993 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:08.087838 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-istio-podinfo\") pod \"e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:08.087993 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:08.087885 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-istio-envoy\") pod \"e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:08.088395 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:08.088117 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-istio-data\") pod \"e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:08.088395 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:08.088198 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-istiod-ca-cert\") pod \"e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:08.090419 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:08.090398 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-istio-envoy\") pod \"e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:08.090519 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:08.090486 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-istio-podinfo\") pod \"e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:08.096789 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:08.096765 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-istio-token\") pod \"e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:08.096906 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:08.096816 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqskj\" (UniqueName: \"kubernetes.io/projected/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-kube-api-access-vqskj\") pod \"e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:08.227254 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:08.227150 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:08.374164 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:08.374127 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr"] Jul 14 18:35:08.377270 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:35:08.377238 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549b4ab0_3fed_49c6_b5a7_b3aba0f2c318.slice/crio-3b695f2e777515bcf4ec797db49ea3044f214c8523b8b8a2b6ef57229c5f78e9 WatchSource:0}: Error finding container 3b695f2e777515bcf4ec797db49ea3044f214c8523b8b8a2b6ef57229c5f78e9: Status 404 returned error can't find the container with id 3b695f2e777515bcf4ec797db49ea3044f214c8523b8b8a2b6ef57229c5f78e9 Jul 14 18:35:08.379462 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:08.379430 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:35:08.379550 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:08.379499 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:35:08.379550 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:08.379528 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:35:08.402407 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:08.402374 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" event={"ID":"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318","Type":"ContainerStarted","Data":"3b695f2e777515bcf4ec797db49ea3044f214c8523b8b8a2b6ef57229c5f78e9"} Jul 14 18:35:09.408173 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:09.408137 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" event={"ID":"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318","Type":"ContainerStarted","Data":"ddb6d41fa9339c422f3235f557b8beca5ee04d4a12e3d37a1d8ecc0c8aa162b1"} Jul 14 18:35:09.434902 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:09.434843 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" podStartSLOduration=2.434821078 podStartE2EDuration="2.434821078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:35:09.43330179 +0000 UTC m=+1514.270645051" watchObservedRunningTime="2026-07-14 18:35:09.434821078 +0000 UTC m=+1514.272164337" Jul 14 18:35:10.227401 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:10.227361 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:10.232431 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:10.232403 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:10.412941 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:10.412908 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:10.413839 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:10.413818 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:14.153990 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:14.153952 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:35:14.217926 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:14.217881 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f37aedf1-8fzrl"] Jul 14 18:35:14.221712 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:14.221677 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f37aedf1-8fzrl" Jul 14 18:35:14.224598 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:14.224570 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 18:35:14.224735 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:14.224571 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9jplk\"" Jul 14 18:35:14.230073 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:14.230050 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f37aedf1-8fzrl"] Jul 14 18:35:14.246873 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:14.246850 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acde68d5-161c-46d6-a261-a8c73dec7739-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f37aedf1-8fzrl\" (UID: \"acde68d5-161c-46d6-a261-a8c73dec7739\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f37aedf1-8fzrl" Jul 14 18:35:14.347430 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:14.347379 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acde68d5-161c-46d6-a261-a8c73dec7739-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f37aedf1-8fzrl\" (UID: \"acde68d5-161c-46d6-a261-a8c73dec7739\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f37aedf1-8fzrl" Jul 14 18:35:14.348090 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:14.348067 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acde68d5-161c-46d6-a261-a8c73dec7739-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f37aedf1-8fzrl\" (UID: \"acde68d5-161c-46d6-a261-a8c73dec7739\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f37aedf1-8fzrl" Jul 14 18:35:14.534503 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:14.534473 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f37aedf1-8fzrl" Jul 14 18:35:14.672173 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:14.672146 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f37aedf1-8fzrl"] Jul 14 18:35:14.674725 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:35:14.674685 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacde68d5_161c_46d6_a261_a8c73dec7739.slice/crio-7f44f6747df023200ca7085de187a777ae28ab7a33bbe2e04c08eaf687e842ab WatchSource:0}: Error finding container 7f44f6747df023200ca7085de187a777ae28ab7a33bbe2e04c08eaf687e842ab: Status 404 returned error can't find the container with id 7f44f6747df023200ca7085de187a777ae28ab7a33bbe2e04c08eaf687e842ab Jul 14 18:35:14.676606 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:14.676589 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 18:35:15.437658 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:15.437617 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f37aedf1-8fzrl" event={"ID":"acde68d5-161c-46d6-a261-a8c73dec7739","Type":"ContainerStarted","Data":"7f44f6747df023200ca7085de187a777ae28ab7a33bbe2e04c08eaf687e842ab"} Jul 14 18:35:18.461868 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:18.461826 2565 generic.go:356] "Generic (PLEG): container finished" podID="acde68d5-161c-46d6-a261-a8c73dec7739" containerID="19c7508af1822ab2d3cf671f0a69978f1e641ec3faad615c8477e2adf47c7ae0" exitCode=0 Jul 14 18:35:18.462308 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:18.461890 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f37aedf1-8fzrl" event={"ID":"acde68d5-161c-46d6-a261-a8c73dec7739","Type":"ContainerDied","Data":"19c7508af1822ab2d3cf671f0a69978f1e641ec3faad615c8477e2adf47c7ae0"} Jul 14 18:35:19.607257 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:19.607230 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f37aedf1-8fzrl" Jul 14 18:35:19.697433 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:19.697398 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acde68d5-161c-46d6-a261-a8c73dec7739-maas-api-service-ca\") pod \"acde68d5-161c-46d6-a261-a8c73dec7739\" (UID: \"acde68d5-161c-46d6-a261-a8c73dec7739\") " Jul 14 18:35:19.697821 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:19.697793 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acde68d5-161c-46d6-a261-a8c73dec7739-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "acde68d5-161c-46d6-a261-a8c73dec7739" (UID: "acde68d5-161c-46d6-a261-a8c73dec7739"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:35:19.797990 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:19.797958 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acde68d5-161c-46d6-a261-a8c73dec7739-maas-api-service-ca\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:35:20.474276 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:20.474243 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f37aedf1-8fzrl" Jul 14 18:35:20.474276 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:20.474262 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f37aedf1-8fzrl" event={"ID":"acde68d5-161c-46d6-a261-a8c73dec7739","Type":"ContainerDied","Data":"7f44f6747df023200ca7085de187a777ae28ab7a33bbe2e04c08eaf687e842ab"} Jul 14 18:35:20.474523 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:20.474328 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f44f6747df023200ca7085de187a777ae28ab7a33bbe2e04c08eaf687e842ab" Jul 14 18:35:22.486676 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:22.486555 2565 generic.go:356] "Generic (PLEG): container finished" podID="549b4ab0-3fed-49c6-b5a7-b3aba0f2c318" containerID="ddb6d41fa9339c422f3235f557b8beca5ee04d4a12e3d37a1d8ecc0c8aa162b1" exitCode=0 Jul 14 18:35:22.486676 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:22.486593 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" event={"ID":"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318","Type":"ContainerDied","Data":"ddb6d41fa9339c422f3235f557b8beca5ee04d4a12e3d37a1d8ecc0c8aa162b1"} Jul 14 18:35:22.487268 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:22.487042 2565 scope.go:117] "RemoveContainer" containerID="ddb6d41fa9339c422f3235f557b8beca5ee04d4a12e3d37a1d8ecc0c8aa162b1" Jul 14 18:35:22.488167 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:22.488139 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:35:22.488274 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:22.488226 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:35:22.488334 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:22.488275 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:35:23.496260 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:23.496212 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" event={"ID":"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318","Type":"ContainerStarted","Data":"95f4cb686663f35c551d1e3f4a526faf68c10183307cf01fc26646bd977648a1"} Jul 14 18:35:24.227844 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:24.227805 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:24.229010 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:24.228980 2565 patch_prober.go:28] interesting pod/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Jul 14 18:35:24.229168 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:24.229041 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" podUID="549b4ab0-3fed-49c6-b5a7-b3aba0f2c318" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Jul 14 18:35:25.227674 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:25.227634 2565 patch_prober.go:28] interesting pod/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Jul 14 18:35:25.228123 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:25.227694 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" podUID="549b4ab0-3fed-49c6-b5a7-b3aba0f2c318" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Jul 14 18:35:25.413252 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:25.413193 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:26.228167 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:26.228094 2565 patch_prober.go:28] interesting pod/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Jul 14 18:35:26.228606 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:26.228199 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" podUID="549b4ab0-3fed-49c6-b5a7-b3aba0f2c318" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Jul 14 18:35:27.227610 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:27.227573 2565 patch_prober.go:28] interesting pod/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Jul 14 18:35:27.227810 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:27.227627 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" podUID="549b4ab0-3fed-49c6-b5a7-b3aba0f2c318" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Jul 14 18:35:27.853896 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:27.853860 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:35:28.232116 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:28.232018 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:28.233184 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:28.233158 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:39.546444 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:39.546406 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr"] Jul 14 18:35:39.546909 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:39.546672 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" podUID="549b4ab0-3fed-49c6-b5a7-b3aba0f2c318" containerName="istio-proxy" containerID="cri-o://95f4cb686663f35c551d1e3f4a526faf68c10183307cf01fc26646bd977648a1" gracePeriod=30 Jul 14 18:35:43.234810 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:43.234768 2565 patch_prober.go:28] interesting pod/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 18:35:43.235237 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:43.234859 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" podUID="549b4ab0-3fed-49c6-b5a7-b3aba0f2c318" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 18:35:44.818169 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:44.818068 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:44.932821 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:44.932782 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-istiod-ca-cert\") pod \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " Jul 14 18:35:44.932997 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:44.932832 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-istio-token\") pod \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " Jul 14 18:35:44.932997 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:44.932860 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-credential-socket\") pod \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " Jul 14 18:35:44.932997 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:44.932901 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-workload-socket\") pod \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " Jul 14 18:35:44.932997 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:44.932942 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-istio-envoy\") pod \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " Jul 14 18:35:44.932997 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:44.932974 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-workload-certs\") pod \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " Jul 14 18:35:44.933288 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:44.933040 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqskj\" (UniqueName: \"kubernetes.io/projected/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-kube-api-access-vqskj\") pod \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " Jul 14 18:35:44.933288 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:44.933066 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-istio-data\") pod \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " Jul 14 18:35:44.933288 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:44.933146 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-istio-podinfo\") pod \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\" (UID: \"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318\") " Jul 14 18:35:44.933288 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:44.933247 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "549b4ab0-3fed-49c6-b5a7-b3aba0f2c318" (UID: "549b4ab0-3fed-49c6-b5a7-b3aba0f2c318"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:35:44.933288 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:44.933266 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "549b4ab0-3fed-49c6-b5a7-b3aba0f2c318" (UID: "549b4ab0-3fed-49c6-b5a7-b3aba0f2c318"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:35:44.933523 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:44.933402 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "549b4ab0-3fed-49c6-b5a7-b3aba0f2c318" (UID: "549b4ab0-3fed-49c6-b5a7-b3aba0f2c318"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:35:44.933523 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:44.933426 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "549b4ab0-3fed-49c6-b5a7-b3aba0f2c318" (UID: "549b4ab0-3fed-49c6-b5a7-b3aba0f2c318"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:35:44.933741 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:44.933719 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-istiod-ca-cert\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:35:44.933921 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:44.933745 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-credential-socket\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:35:44.933921 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:44.933759 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-workload-socket\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:35:44.933921 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:44.933774 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-workload-certs\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:35:44.934167 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:44.934078 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-istio-data" (OuterVolumeSpecName: "istio-data") pod "549b4ab0-3fed-49c6-b5a7-b3aba0f2c318" (UID: "549b4ab0-3fed-49c6-b5a7-b3aba0f2c318"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:35:44.935442 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:44.935416 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-istio-token" (OuterVolumeSpecName: "istio-token") pod "549b4ab0-3fed-49c6-b5a7-b3aba0f2c318" (UID: "549b4ab0-3fed-49c6-b5a7-b3aba0f2c318"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:35:44.935599 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:44.935581 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "549b4ab0-3fed-49c6-b5a7-b3aba0f2c318" (UID: "549b4ab0-3fed-49c6-b5a7-b3aba0f2c318"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 18:35:44.935899 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:44.935875 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "549b4ab0-3fed-49c6-b5a7-b3aba0f2c318" (UID: "549b4ab0-3fed-49c6-b5a7-b3aba0f2c318"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:35:44.935967 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:44.935875 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-kube-api-access-vqskj" (OuterVolumeSpecName: "kube-api-access-vqskj") pod "549b4ab0-3fed-49c6-b5a7-b3aba0f2c318" (UID: "549b4ab0-3fed-49c6-b5a7-b3aba0f2c318"). InnerVolumeSpecName "kube-api-access-vqskj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:35:45.034666 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.034625 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-istio-token\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:35:45.034666 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.034656 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-istio-envoy\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:35:45.034666 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.034666 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqskj\" (UniqueName: \"kubernetes.io/projected/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-kube-api-access-vqskj\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:35:45.034666 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.034678 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-istio-data\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:35:45.034940 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.034687 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318-istio-podinfo\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:35:45.609121 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.609062 2565 generic.go:356] "Generic (PLEG): container finished" podID="549b4ab0-3fed-49c6-b5a7-b3aba0f2c318" containerID="95f4cb686663f35c551d1e3f4a526faf68c10183307cf01fc26646bd977648a1" exitCode=0 Jul 14 18:35:45.609320 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.609145 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" Jul 14 18:35:45.609320 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.609163 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" event={"ID":"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318","Type":"ContainerDied","Data":"95f4cb686663f35c551d1e3f4a526faf68c10183307cf01fc26646bd977648a1"} Jul 14 18:35:45.609320 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.609220 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr" event={"ID":"549b4ab0-3fed-49c6-b5a7-b3aba0f2c318","Type":"ContainerDied","Data":"3b695f2e777515bcf4ec797db49ea3044f214c8523b8b8a2b6ef57229c5f78e9"} Jul 14 18:35:45.609320 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.609242 2565 scope.go:117] "RemoveContainer" containerID="95f4cb686663f35c551d1e3f4a526faf68c10183307cf01fc26646bd977648a1" Jul 14 18:35:45.618397 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.618377 2565 scope.go:117] "RemoveContainer" containerID="ddb6d41fa9339c422f3235f557b8beca5ee04d4a12e3d37a1d8ecc0c8aa162b1" Jul 14 18:35:45.627325 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.627303 2565 scope.go:117] "RemoveContainer" containerID="95f4cb686663f35c551d1e3f4a526faf68c10183307cf01fc26646bd977648a1" Jul 14 18:35:45.627680 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:35:45.627655 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95f4cb686663f35c551d1e3f4a526faf68c10183307cf01fc26646bd977648a1\": container with ID starting with 95f4cb686663f35c551d1e3f4a526faf68c10183307cf01fc26646bd977648a1 not found: ID does not exist" containerID="95f4cb686663f35c551d1e3f4a526faf68c10183307cf01fc26646bd977648a1" Jul 14 18:35:45.627762 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.627695 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95f4cb686663f35c551d1e3f4a526faf68c10183307cf01fc26646bd977648a1"} err="failed to get container status \"95f4cb686663f35c551d1e3f4a526faf68c10183307cf01fc26646bd977648a1\": rpc error: code = NotFound desc = could not find container \"95f4cb686663f35c551d1e3f4a526faf68c10183307cf01fc26646bd977648a1\": container with ID starting with 95f4cb686663f35c551d1e3f4a526faf68c10183307cf01fc26646bd977648a1 not found: ID does not exist" Jul 14 18:35:45.627762 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.627722 2565 scope.go:117] "RemoveContainer" containerID="ddb6d41fa9339c422f3235f557b8beca5ee04d4a12e3d37a1d8ecc0c8aa162b1" Jul 14 18:35:45.628014 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:35:45.627995 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddb6d41fa9339c422f3235f557b8beca5ee04d4a12e3d37a1d8ecc0c8aa162b1\": container with ID starting with ddb6d41fa9339c422f3235f557b8beca5ee04d4a12e3d37a1d8ecc0c8aa162b1 not found: ID does not exist" containerID="ddb6d41fa9339c422f3235f557b8beca5ee04d4a12e3d37a1d8ecc0c8aa162b1" Jul 14 18:35:45.628056 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.628020 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddb6d41fa9339c422f3235f557b8beca5ee04d4a12e3d37a1d8ecc0c8aa162b1"} err="failed to get container status \"ddb6d41fa9339c422f3235f557b8beca5ee04d4a12e3d37a1d8ecc0c8aa162b1\": rpc error: code = NotFound desc = could not find container \"ddb6d41fa9339c422f3235f557b8beca5ee04d4a12e3d37a1d8ecc0c8aa162b1\": container with ID starting with ddb6d41fa9339c422f3235f557b8beca5ee04d4a12e3d37a1d8ecc0c8aa162b1 not found: ID does not exist" Jul 14 18:35:45.632643 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.632609 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr"] Jul 14 18:35:45.637330 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.637300 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f37aedf1-openshift-default-654c94f669-kp8wr"] Jul 14 18:35:45.692654 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.692618 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="549b4ab0-3fed-49c6-b5a7-b3aba0f2c318" path="/var/lib/kubelet/pods/549b4ab0-3fed-49c6-b5a7-b3aba0f2c318/volumes" Jul 14 18:35:45.984857 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.984771 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-7dcmr"] Jul 14 18:35:45.985285 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.985133 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="549b4ab0-3fed-49c6-b5a7-b3aba0f2c318" containerName="istio-proxy" Jul 14 18:35:45.985285 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.985146 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="549b4ab0-3fed-49c6-b5a7-b3aba0f2c318" containerName="istio-proxy" Jul 14 18:35:45.985285 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.985157 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="549b4ab0-3fed-49c6-b5a7-b3aba0f2c318" containerName="istio-proxy" Jul 14 18:35:45.985285 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.985162 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="549b4ab0-3fed-49c6-b5a7-b3aba0f2c318" containerName="istio-proxy" Jul 14 18:35:45.985285 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.985175 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acde68d5-161c-46d6-a261-a8c73dec7739" containerName="revoke-keys" Jul 14 18:35:45.985285 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.985180 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="acde68d5-161c-46d6-a261-a8c73dec7739" containerName="revoke-keys" Jul 14 18:35:45.985285 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.985242 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="549b4ab0-3fed-49c6-b5a7-b3aba0f2c318" containerName="istio-proxy" Jul 14 18:35:45.985285 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.985251 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="549b4ab0-3fed-49c6-b5a7-b3aba0f2c318" containerName="istio-proxy" Jul 14 18:35:45.985285 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.985257 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="acde68d5-161c-46d6-a261-a8c73dec7739" containerName="revoke-keys" Jul 14 18:35:45.990050 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.990029 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-7dcmr" Jul 14 18:35:45.992994 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.992968 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 18:35:45.993149 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.993032 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9jplk\"" Jul 14 18:35:45.998422 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:45.998399 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-7dcmr"] Jul 14 18:35:46.145234 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:46.145202 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-499cb51a-7dcmr\" (UID: \"e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-7dcmr" Jul 14 18:35:46.246652 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:46.246558 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-499cb51a-7dcmr\" (UID: \"e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-7dcmr" Jul 14 18:35:46.247273 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:46.247252 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-499cb51a-7dcmr\" (UID: \"e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-7dcmr" Jul 14 18:35:46.303644 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:46.303604 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-7dcmr" Jul 14 18:35:46.446897 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:46.446870 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-7dcmr"] Jul 14 18:35:46.449205 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:35:46.449166 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e252e4_8d07_4dfc_81c5_60b7b8c4d7b4.slice/crio-5d064b75cee93e255e374fd58651a88e5dcdf6762190b3333c0c09f5c2bf6dd1 WatchSource:0}: Error finding container 5d064b75cee93e255e374fd58651a88e5dcdf6762190b3333c0c09f5c2bf6dd1: Status 404 returned error can't find the container with id 5d064b75cee93e255e374fd58651a88e5dcdf6762190b3333c0c09f5c2bf6dd1 Jul 14 18:35:46.616034 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:46.615997 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-7dcmr" event={"ID":"e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4","Type":"ContainerStarted","Data":"5d064b75cee93e255e374fd58651a88e5dcdf6762190b3333c0c09f5c2bf6dd1"} Jul 14 18:35:47.622532 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:47.622499 2565 generic.go:356] "Generic (PLEG): container finished" podID="e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4" containerID="ec999894f721413c4323a9c53d07bc6dfed21a6b3f897b4566491c0e8cef93f2" exitCode=7 Jul 14 18:35:47.622927 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:47.622551 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-7dcmr" event={"ID":"e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4","Type":"ContainerDied","Data":"ec999894f721413c4323a9c53d07bc6dfed21a6b3f897b4566491c0e8cef93f2"} Jul 14 18:35:47.622927 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:47.622789 2565 scope.go:117] "RemoveContainer" containerID="ec999894f721413c4323a9c53d07bc6dfed21a6b3f897b4566491c0e8cef93f2" Jul 14 18:35:48.628927 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:48.628832 2565 generic.go:356] "Generic (PLEG): container finished" podID="e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4" containerID="86d7a16ed0ca2e2821507366922518ef2a51a2383c2740aef7ae6378eb716155" exitCode=7 Jul 14 18:35:48.628927 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:48.628915 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-7dcmr" event={"ID":"e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4","Type":"ContainerDied","Data":"86d7a16ed0ca2e2821507366922518ef2a51a2383c2740aef7ae6378eb716155"} Jul 14 18:35:48.629413 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:48.628978 2565 scope.go:117] "RemoveContainer" containerID="ec999894f721413c4323a9c53d07bc6dfed21a6b3f897b4566491c0e8cef93f2" Jul 14 18:35:48.629413 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:48.629221 2565 scope.go:117] "RemoveContainer" containerID="86d7a16ed0ca2e2821507366922518ef2a51a2383c2740aef7ae6378eb716155" Jul 14 18:35:48.629495 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:35:48.629419 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-499cb51a-7dcmr_odh-ai-gateway-infra(e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-7dcmr" podUID="e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4" Jul 14 18:35:48.758255 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:48.758221 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:35:49.634691 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:35:49.634660 2565 scope.go:117] "RemoveContainer" containerID="86d7a16ed0ca2e2821507366922518ef2a51a2383c2740aef7ae6378eb716155" Jul 14 18:35:49.635116 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:35:49.634849 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-499cb51a-7dcmr_odh-ai-gateway-infra(e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-7dcmr" podUID="e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4" Jul 14 18:36:01.684699 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:01.684666 2565 scope.go:117] "RemoveContainer" containerID="86d7a16ed0ca2e2821507366922518ef2a51a2383c2740aef7ae6378eb716155" Jul 14 18:36:02.696407 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:02.696375 2565 generic.go:356] "Generic (PLEG): container finished" podID="e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4" containerID="299bae6d501dd4745a1a6af764b452682f5d0976c5b36594587a528fc5ad0adf" exitCode=0 Jul 14 18:36:02.696814 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:02.696441 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-7dcmr" event={"ID":"e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4","Type":"ContainerDied","Data":"299bae6d501dd4745a1a6af764b452682f5d0976c5b36594587a528fc5ad0adf"} Jul 14 18:36:02.696814 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:02.696502 2565 scope.go:117] "RemoveContainer" containerID="86d7a16ed0ca2e2821507366922518ef2a51a2383c2740aef7ae6378eb716155" Jul 14 18:36:03.730501 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:03.730463 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-7dcmr"] Jul 14 18:36:03.844798 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:03.844768 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-7dcmr" Jul 14 18:36:03.897986 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:03.897949 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4-maas-api-service-ca\") pod \"e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4\" (UID: \"e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4\") " Jul 14 18:36:03.898362 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:03.898337 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4" (UID: "e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:36:03.999535 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:03.999443 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4-maas-api-service-ca\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:36:04.708825 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:04.708795 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-7dcmr" Jul 14 18:36:04.709081 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:04.708839 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-7dcmr" event={"ID":"e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4","Type":"ContainerDied","Data":"5d064b75cee93e255e374fd58651a88e5dcdf6762190b3333c0c09f5c2bf6dd1"} Jul 14 18:36:04.709081 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:04.708888 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d064b75cee93e255e374fd58651a88e5dcdf6762190b3333c0c09f5c2bf6dd1" Jul 14 18:36:04.747485 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:04.747451 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-7dcmr"] Jul 14 18:36:04.752009 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:04.751982 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-7dcmr"] Jul 14 18:36:05.690332 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:05.690294 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4" path="/var/lib/kubelet/pods/e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4/volumes" Jul 14 18:36:36.041610 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:36.041568 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-gk9lz"] Jul 14 18:36:36.042137 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:36.042020 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4" containerName="revoke-keys" Jul 14 18:36:36.042137 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:36.042038 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4" containerName="revoke-keys" Jul 14 18:36:36.042137 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:36.042078 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4" containerName="revoke-keys" Jul 14 18:36:36.042137 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:36.042086 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4" containerName="revoke-keys" Jul 14 18:36:36.042276 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:36.042178 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4" containerName="revoke-keys" Jul 14 18:36:36.042276 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:36.042195 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4" containerName="revoke-keys" Jul 14 18:36:36.045153 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:36.045126 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-gk9lz" Jul 14 18:36:36.048027 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:36.048004 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9jplk\"" Jul 14 18:36:36.048165 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:36.048038 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 18:36:36.054971 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:36.054946 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-gk9lz"] Jul 14 18:36:36.182746 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:36.182708 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e8e2ff3-29b7-41ba-99ba-aafaedc1e795-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-499cb51a-gk9lz\" (UID: \"6e8e2ff3-29b7-41ba-99ba-aafaedc1e795\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-gk9lz" Jul 14 18:36:36.283691 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:36.283655 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e8e2ff3-29b7-41ba-99ba-aafaedc1e795-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-499cb51a-gk9lz\" (UID: \"6e8e2ff3-29b7-41ba-99ba-aafaedc1e795\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-gk9lz" Jul 14 18:36:36.284350 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:36.284328 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e8e2ff3-29b7-41ba-99ba-aafaedc1e795-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-499cb51a-gk9lz\" (UID: \"6e8e2ff3-29b7-41ba-99ba-aafaedc1e795\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-gk9lz" Jul 14 18:36:36.358136 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:36.358036 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-gk9lz" Jul 14 18:36:36.494005 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:36.493971 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-gk9lz"] Jul 14 18:36:36.497989 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:36:36.497961 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e8e2ff3_29b7_41ba_99ba_aafaedc1e795.slice/crio-d1d05261775486b7b8ed9be95da0316a889d1e445938bfcf5f3f0e20b5fa1c0b WatchSource:0}: Error finding container d1d05261775486b7b8ed9be95da0316a889d1e445938bfcf5f3f0e20b5fa1c0b: Status 404 returned error can't find the container with id d1d05261775486b7b8ed9be95da0316a889d1e445938bfcf5f3f0e20b5fa1c0b Jul 14 18:36:36.857724 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:36.857689 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-gk9lz" event={"ID":"6e8e2ff3-29b7-41ba-99ba-aafaedc1e795","Type":"ContainerStarted","Data":"a79c298a1c9595f9e2b132f02760e75818947e45716eda0ec8da0dab1e55066b"} Jul 14 18:36:36.857933 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:36.857749 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-gk9lz" event={"ID":"6e8e2ff3-29b7-41ba-99ba-aafaedc1e795","Type":"ContainerStarted","Data":"d1d05261775486b7b8ed9be95da0316a889d1e445938bfcf5f3f0e20b5fa1c0b"} Jul 14 18:36:36.878473 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:36.878420 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-gk9lz" podStartSLOduration=0.878402005 podStartE2EDuration="878.402005ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:36:36.876917846 +0000 UTC m=+1601.714261103" watchObservedRunningTime="2026-07-14 18:36:36.878402005 +0000 UTC m=+1601.715745304" Jul 14 18:36:37.863969 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:37.863928 2565 generic.go:356] "Generic (PLEG): container finished" podID="6e8e2ff3-29b7-41ba-99ba-aafaedc1e795" containerID="a79c298a1c9595f9e2b132f02760e75818947e45716eda0ec8da0dab1e55066b" exitCode=0 Jul 14 18:36:37.864375 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:37.864021 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-gk9lz" event={"ID":"6e8e2ff3-29b7-41ba-99ba-aafaedc1e795","Type":"ContainerDied","Data":"a79c298a1c9595f9e2b132f02760e75818947e45716eda0ec8da0dab1e55066b"} Jul 14 18:36:39.009927 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:39.009894 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-gk9lz" Jul 14 18:36:39.110347 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:39.110308 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e8e2ff3-29b7-41ba-99ba-aafaedc1e795-maas-api-service-ca\") pod \"6e8e2ff3-29b7-41ba-99ba-aafaedc1e795\" (UID: \"6e8e2ff3-29b7-41ba-99ba-aafaedc1e795\") " Jul 14 18:36:39.110716 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:39.110691 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e8e2ff3-29b7-41ba-99ba-aafaedc1e795-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6e8e2ff3-29b7-41ba-99ba-aafaedc1e795" (UID: "6e8e2ff3-29b7-41ba-99ba-aafaedc1e795"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:36:39.211158 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:39.211070 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e8e2ff3-29b7-41ba-99ba-aafaedc1e795-maas-api-service-ca\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:36:39.874793 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:39.874760 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-gk9lz" Jul 14 18:36:39.874793 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:39.874772 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-499cb51a-gk9lz" event={"ID":"6e8e2ff3-29b7-41ba-99ba-aafaedc1e795","Type":"ContainerDied","Data":"d1d05261775486b7b8ed9be95da0316a889d1e445938bfcf5f3f0e20b5fa1c0b"} Jul 14 18:36:39.874987 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:39.874824 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1d05261775486b7b8ed9be95da0316a889d1e445938bfcf5f3f0e20b5fa1c0b" Jul 14 18:36:49.056228 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:49.056148 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:36:53.220972 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.220933 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf"] Jul 14 18:36:53.221566 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.221543 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4" containerName="revoke-keys" Jul 14 18:36:53.221641 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.221570 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4" containerName="revoke-keys" Jul 14 18:36:53.221641 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.221599 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e8e2ff3-29b7-41ba-99ba-aafaedc1e795" containerName="revoke-keys" Jul 14 18:36:53.221641 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.221608 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e8e2ff3-29b7-41ba-99ba-aafaedc1e795" containerName="revoke-keys" Jul 14 18:36:53.221784 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.221718 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5e252e4-8d07-4dfc-81c5-60b7b8c4d7b4" containerName="revoke-keys" Jul 14 18:36:53.221784 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.221731 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e8e2ff3-29b7-41ba-99ba-aafaedc1e795" containerName="revoke-keys" Jul 14 18:36:53.225364 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.225335 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:53.228177 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.228155 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1808df45-openshift-default-dockercfg-szvbv\"" Jul 14 18:36:53.241407 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.241376 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf"] Jul 14 18:36:53.317417 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.317378 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c95bd58-148f-4245-b5d2-6d6b72893c8a-credential-socket\") pod \"e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:53.317417 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.317418 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c95bd58-148f-4245-b5d2-6d6b72893c8a-istio-token\") pod \"e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:53.317654 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.317475 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c95bd58-148f-4245-b5d2-6d6b72893c8a-istiod-ca-cert\") pod \"e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:53.317654 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.317495 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwgbp\" (UniqueName: \"kubernetes.io/projected/8c95bd58-148f-4245-b5d2-6d6b72893c8a-kube-api-access-bwgbp\") pod \"e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:53.317654 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.317516 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c95bd58-148f-4245-b5d2-6d6b72893c8a-istio-envoy\") pod \"e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:53.317654 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.317531 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c95bd58-148f-4245-b5d2-6d6b72893c8a-istio-data\") pod \"e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:53.317654 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.317563 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c95bd58-148f-4245-b5d2-6d6b72893c8a-workload-socket\") pod \"e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:53.317654 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.317577 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c95bd58-148f-4245-b5d2-6d6b72893c8a-workload-certs\") pod \"e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:53.317654 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.317631 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c95bd58-148f-4245-b5d2-6d6b72893c8a-istio-podinfo\") pod \"e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:53.418649 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.418606 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c95bd58-148f-4245-b5d2-6d6b72893c8a-istio-podinfo\") pod \"e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:53.418649 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.418653 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c95bd58-148f-4245-b5d2-6d6b72893c8a-credential-socket\") pod \"e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:53.418917 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.418680 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c95bd58-148f-4245-b5d2-6d6b72893c8a-istio-token\") pod \"e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:53.418917 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.418738 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c95bd58-148f-4245-b5d2-6d6b72893c8a-istiod-ca-cert\") pod \"e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:53.418917 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.418765 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwgbp\" (UniqueName: \"kubernetes.io/projected/8c95bd58-148f-4245-b5d2-6d6b72893c8a-kube-api-access-bwgbp\") pod \"e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:53.418917 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.418797 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c95bd58-148f-4245-b5d2-6d6b72893c8a-istio-envoy\") pod \"e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:53.418917 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.418861 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c95bd58-148f-4245-b5d2-6d6b72893c8a-istio-data\") pod \"e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:53.419227 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.418955 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c95bd58-148f-4245-b5d2-6d6b72893c8a-workload-socket\") pod \"e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:53.419227 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.418986 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c95bd58-148f-4245-b5d2-6d6b72893c8a-workload-certs\") pod \"e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:53.419227 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.419093 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c95bd58-148f-4245-b5d2-6d6b72893c8a-credential-socket\") pod \"e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:53.419227 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.419219 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c95bd58-148f-4245-b5d2-6d6b72893c8a-istio-data\") pod \"e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:53.419424 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.419297 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c95bd58-148f-4245-b5d2-6d6b72893c8a-workload-socket\") pod \"e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:53.419424 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.419371 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c95bd58-148f-4245-b5d2-6d6b72893c8a-workload-certs\") pod \"e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:53.419690 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.419614 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c95bd58-148f-4245-b5d2-6d6b72893c8a-istiod-ca-cert\") pod \"e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:53.421285 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.421259 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c95bd58-148f-4245-b5d2-6d6b72893c8a-istio-envoy\") pod \"e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:53.421444 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.421426 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c95bd58-148f-4245-b5d2-6d6b72893c8a-istio-podinfo\") pod \"e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:53.429003 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.428961 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c95bd58-148f-4245-b5d2-6d6b72893c8a-istio-token\") pod \"e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:53.429162 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.429011 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwgbp\" (UniqueName: \"kubernetes.io/projected/8c95bd58-148f-4245-b5d2-6d6b72893c8a-kube-api-access-bwgbp\") pod \"e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:53.537423 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.537384 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:53.691296 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.691271 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf"] Jul 14 18:36:53.692335 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.692297 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:36:53.692414 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.692382 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:36:53.692451 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.692423 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:36:53.949244 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.949152 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" event={"ID":"8c95bd58-148f-4245-b5d2-6d6b72893c8a","Type":"ContainerStarted","Data":"91e0d73ba2ff83c014391db0021cd6c1f4ba55d27454293a8433107be3266991"} Jul 14 18:36:53.949244 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.949225 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" event={"ID":"8c95bd58-148f-4245-b5d2-6d6b72893c8a","Type":"ContainerStarted","Data":"94c41da9fb4f0d4407fe8ca8a71df96ad33c7ba3209d90cfd5a67b2ec40c05f3"} Jul 14 18:36:53.974824 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:53.974574 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" podStartSLOduration=0.97455141 podStartE2EDuration="974.55141ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:36:53.973900803 +0000 UTC m=+1618.811244063" watchObservedRunningTime="2026-07-14 18:36:53.97455141 +0000 UTC m=+1618.811894672" Jul 14 18:36:54.537717 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:54.537678 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:54.543633 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:54.543590 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:54.953732 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:54.953650 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:54.954740 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:54.954714 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:36:58.255457 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:58.255420 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:36:59.564298 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:59.564261 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1808df45-4tx94"] Jul 14 18:36:59.567655 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:59.567638 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1808df45-4tx94" Jul 14 18:36:59.570446 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:59.570415 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9jplk\"" Jul 14 18:36:59.570446 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:59.570434 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 18:36:59.576124 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:59.576073 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1808df45-4tx94"] Jul 14 18:36:59.678770 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:59.678717 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b965b04-799c-4da6-96e2-8c5de94a548e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1808df45-4tx94\" (UID: \"3b965b04-799c-4da6-96e2-8c5de94a548e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1808df45-4tx94" Jul 14 18:36:59.780219 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:59.780183 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b965b04-799c-4da6-96e2-8c5de94a548e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1808df45-4tx94\" (UID: \"3b965b04-799c-4da6-96e2-8c5de94a548e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1808df45-4tx94" Jul 14 18:36:59.780939 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:59.780920 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b965b04-799c-4da6-96e2-8c5de94a548e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1808df45-4tx94\" (UID: \"3b965b04-799c-4da6-96e2-8c5de94a548e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1808df45-4tx94" Jul 14 18:36:59.878614 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:36:59.878524 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1808df45-4tx94" Jul 14 18:37:00.066658 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:00.066626 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1808df45-4tx94"] Jul 14 18:37:00.070128 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:37:00.070089 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b965b04_799c_4da6_96e2_8c5de94a548e.slice/crio-2782b2deebda1e03e9c3373dc1b472ffae582ccba269a84595cf0d5dcd282355 WatchSource:0}: Error finding container 2782b2deebda1e03e9c3373dc1b472ffae582ccba269a84595cf0d5dcd282355: Status 404 returned error can't find the container with id 2782b2deebda1e03e9c3373dc1b472ffae582ccba269a84595cf0d5dcd282355 Jul 14 18:37:00.982900 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:00.982856 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1808df45-4tx94" event={"ID":"3b965b04-799c-4da6-96e2-8c5de94a548e","Type":"ContainerStarted","Data":"4896532bd7e99c2e777446374d9e164578dea642b0ee729ba91a04d07d831b03"} Jul 14 18:37:00.983317 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:00.982921 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1808df45-4tx94" event={"ID":"3b965b04-799c-4da6-96e2-8c5de94a548e","Type":"ContainerStarted","Data":"2782b2deebda1e03e9c3373dc1b472ffae582ccba269a84595cf0d5dcd282355"} Jul 14 18:37:01.991693 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:01.991657 2565 generic.go:356] "Generic (PLEG): container finished" podID="3b965b04-799c-4da6-96e2-8c5de94a548e" containerID="4896532bd7e99c2e777446374d9e164578dea642b0ee729ba91a04d07d831b03" exitCode=7 Jul 14 18:37:01.992684 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:01.992336 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1808df45-4tx94" event={"ID":"3b965b04-799c-4da6-96e2-8c5de94a548e","Type":"ContainerDied","Data":"4896532bd7e99c2e777446374d9e164578dea642b0ee729ba91a04d07d831b03"} Jul 14 18:37:01.992872 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:01.992616 2565 scope.go:117] "RemoveContainer" containerID="4896532bd7e99c2e777446374d9e164578dea642b0ee729ba91a04d07d831b03" Jul 14 18:37:03.002713 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:03.002651 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1808df45-4tx94" event={"ID":"3b965b04-799c-4da6-96e2-8c5de94a548e","Type":"ContainerStarted","Data":"afd0482afc67956b7b68c7c4c78064ad3bbe5a98798c649dd14c9348b602604a"} Jul 14 18:37:03.024354 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:03.024269 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1808df45-4tx94" podStartSLOduration=4.024250182 podStartE2EDuration="4.024250182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:36:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:37:01.004415137 +0000 UTC m=+1625.841758396" watchObservedRunningTime="2026-07-14 18:37:03.024250182 +0000 UTC m=+1627.861593439" Jul 14 18:37:04.010226 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:04.010136 2565 generic.go:356] "Generic (PLEG): container finished" podID="3b965b04-799c-4da6-96e2-8c5de94a548e" containerID="afd0482afc67956b7b68c7c4c78064ad3bbe5a98798c649dd14c9348b602604a" exitCode=0 Jul 14 18:37:04.010613 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:04.010216 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1808df45-4tx94" event={"ID":"3b965b04-799c-4da6-96e2-8c5de94a548e","Type":"ContainerDied","Data":"afd0482afc67956b7b68c7c4c78064ad3bbe5a98798c649dd14c9348b602604a"} Jul 14 18:37:04.010613 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:04.010285 2565 scope.go:117] "RemoveContainer" containerID="4896532bd7e99c2e777446374d9e164578dea642b0ee729ba91a04d07d831b03" Jul 14 18:37:05.165308 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:05.165282 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1808df45-4tx94" Jul 14 18:37:05.237293 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:05.237252 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b965b04-799c-4da6-96e2-8c5de94a548e-maas-api-service-ca\") pod \"3b965b04-799c-4da6-96e2-8c5de94a548e\" (UID: \"3b965b04-799c-4da6-96e2-8c5de94a548e\") " Jul 14 18:37:05.237618 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:05.237596 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b965b04-799c-4da6-96e2-8c5de94a548e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3b965b04-799c-4da6-96e2-8c5de94a548e" (UID: "3b965b04-799c-4da6-96e2-8c5de94a548e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:37:05.338604 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:05.338518 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b965b04-799c-4da6-96e2-8c5de94a548e-maas-api-service-ca\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:37:06.022652 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:06.022608 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1808df45-4tx94" event={"ID":"3b965b04-799c-4da6-96e2-8c5de94a548e","Type":"ContainerDied","Data":"2782b2deebda1e03e9c3373dc1b472ffae582ccba269a84595cf0d5dcd282355"} Jul 14 18:37:06.024688 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:06.022698 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2782b2deebda1e03e9c3373dc1b472ffae582ccba269a84595cf0d5dcd282355" Jul 14 18:37:06.024688 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:06.022632 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1808df45-4tx94" Jul 14 18:37:07.029719 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:07.029686 2565 generic.go:356] "Generic (PLEG): container finished" podID="8c95bd58-148f-4245-b5d2-6d6b72893c8a" containerID="91e0d73ba2ff83c014391db0021cd6c1f4ba55d27454293a8433107be3266991" exitCode=0 Jul 14 18:37:07.030140 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:07.029767 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" event={"ID":"8c95bd58-148f-4245-b5d2-6d6b72893c8a","Type":"ContainerDied","Data":"91e0d73ba2ff83c014391db0021cd6c1f4ba55d27454293a8433107be3266991"} Jul 14 18:37:07.030206 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:07.030191 2565 scope.go:117] "RemoveContainer" containerID="91e0d73ba2ff83c014391db0021cd6c1f4ba55d27454293a8433107be3266991" Jul 14 18:37:07.031174 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:07.031149 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:37:07.031264 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:07.031211 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:37:07.031264 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:07.031242 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:37:08.036618 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:08.036583 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" event={"ID":"8c95bd58-148f-4245-b5d2-6d6b72893c8a","Type":"ContainerStarted","Data":"58d909b7f51556c01d2a8badf64a57758a16ecc0ddaed00ed02fc7f8d0f1227a"} Jul 14 18:37:08.537812 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:08.537777 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:37:08.538902 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:08.538881 2565 patch_prober.go:28] interesting pod/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 14 18:37:08.538987 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:08.538931 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" podUID="8c95bd58-148f-4245-b5d2-6d6b72893c8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 14 18:37:09.537902 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:09.537865 2565 patch_prober.go:28] interesting pod/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 14 18:37:09.538412 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:09.537926 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" podUID="8c95bd58-148f-4245-b5d2-6d6b72893c8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 14 18:37:09.953640 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:09.953552 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:37:10.537950 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:10.537909 2565 patch_prober.go:28] interesting pod/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 14 18:37:10.538377 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:10.537965 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" podUID="8c95bd58-148f-4245-b5d2-6d6b72893c8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 14 18:37:11.538507 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:11.538468 2565 patch_prober.go:28] interesting pod/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 14 18:37:11.538925 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:11.538524 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" podUID="8c95bd58-148f-4245-b5d2-6d6b72893c8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 14 18:37:12.542213 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:12.542185 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:37:12.543092 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:12.543075 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:37:16.053896 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:16.053860 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:37:25.728647 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:25.728606 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf"] Jul 14 18:37:25.729173 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:25.728826 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" podUID="8c95bd58-148f-4245-b5d2-6d6b72893c8a" containerName="istio-proxy" containerID="cri-o://58d909b7f51556c01d2a8badf64a57758a16ecc0ddaed00ed02fc7f8d0f1227a" gracePeriod=30 Jul 14 18:37:27.544988 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:27.544948 2565 patch_prober.go:28] interesting pod/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 18:37:27.545385 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:27.545024 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" podUID="8c95bd58-148f-4245-b5d2-6d6b72893c8a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 18:37:30.986370 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:30.986335 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:37:31.085363 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.085330 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c95bd58-148f-4245-b5d2-6d6b72893c8a-istio-token\") pod \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " Jul 14 18:37:31.085363 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.085367 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwgbp\" (UniqueName: \"kubernetes.io/projected/8c95bd58-148f-4245-b5d2-6d6b72893c8a-kube-api-access-bwgbp\") pod \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " Jul 14 18:37:31.085605 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.085461 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c95bd58-148f-4245-b5d2-6d6b72893c8a-istio-envoy\") pod \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " Jul 14 18:37:31.085605 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.085487 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c95bd58-148f-4245-b5d2-6d6b72893c8a-istio-data\") pod \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " Jul 14 18:37:31.085715 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.085626 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c95bd58-148f-4245-b5d2-6d6b72893c8a-workload-certs\") pod \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " Jul 14 18:37:31.085715 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.085668 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c95bd58-148f-4245-b5d2-6d6b72893c8a-credential-socket\") pod \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " Jul 14 18:37:31.085715 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.085697 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c95bd58-148f-4245-b5d2-6d6b72893c8a-istiod-ca-cert\") pod \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " Jul 14 18:37:31.085862 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.085728 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c95bd58-148f-4245-b5d2-6d6b72893c8a-workload-socket\") pod \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " Jul 14 18:37:31.085862 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.085771 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c95bd58-148f-4245-b5d2-6d6b72893c8a-istio-podinfo\") pod \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\" (UID: \"8c95bd58-148f-4245-b5d2-6d6b72893c8a\") " Jul 14 18:37:31.085987 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.085949 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c95bd58-148f-4245-b5d2-6d6b72893c8a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8c95bd58-148f-4245-b5d2-6d6b72893c8a" (UID: "8c95bd58-148f-4245-b5d2-6d6b72893c8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:37:31.086080 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.086050 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c95bd58-148f-4245-b5d2-6d6b72893c8a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8c95bd58-148f-4245-b5d2-6d6b72893c8a" (UID: "8c95bd58-148f-4245-b5d2-6d6b72893c8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:37:31.086181 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.086144 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c95bd58-148f-4245-b5d2-6d6b72893c8a-credential-socket\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:37:31.086181 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.086160 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c95bd58-148f-4245-b5d2-6d6b72893c8a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8c95bd58-148f-4245-b5d2-6d6b72893c8a" (UID: "8c95bd58-148f-4245-b5d2-6d6b72893c8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:37:31.086322 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.086299 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c95bd58-148f-4245-b5d2-6d6b72893c8a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8c95bd58-148f-4245-b5d2-6d6b72893c8a" (UID: "8c95bd58-148f-4245-b5d2-6d6b72893c8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:37:31.087076 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.087049 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c95bd58-148f-4245-b5d2-6d6b72893c8a-istio-data" (OuterVolumeSpecName: "istio-data") pod "8c95bd58-148f-4245-b5d2-6d6b72893c8a" (UID: "8c95bd58-148f-4245-b5d2-6d6b72893c8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:37:31.088295 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.088270 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c95bd58-148f-4245-b5d2-6d6b72893c8a-kube-api-access-bwgbp" (OuterVolumeSpecName: "kube-api-access-bwgbp") pod "8c95bd58-148f-4245-b5d2-6d6b72893c8a" (UID: "8c95bd58-148f-4245-b5d2-6d6b72893c8a"). InnerVolumeSpecName "kube-api-access-bwgbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:37:31.088435 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.088413 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c95bd58-148f-4245-b5d2-6d6b72893c8a-istio-token" (OuterVolumeSpecName: "istio-token") pod "8c95bd58-148f-4245-b5d2-6d6b72893c8a" (UID: "8c95bd58-148f-4245-b5d2-6d6b72893c8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:37:31.088713 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.088691 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c95bd58-148f-4245-b5d2-6d6b72893c8a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8c95bd58-148f-4245-b5d2-6d6b72893c8a" (UID: "8c95bd58-148f-4245-b5d2-6d6b72893c8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:37:31.088798 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.088744 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c95bd58-148f-4245-b5d2-6d6b72893c8a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8c95bd58-148f-4245-b5d2-6d6b72893c8a" (UID: "8c95bd58-148f-4245-b5d2-6d6b72893c8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 18:37:31.150193 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.150161 2565 generic.go:356] "Generic (PLEG): container finished" podID="8c95bd58-148f-4245-b5d2-6d6b72893c8a" containerID="58d909b7f51556c01d2a8badf64a57758a16ecc0ddaed00ed02fc7f8d0f1227a" exitCode=0 Jul 14 18:37:31.150460 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.150208 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" Jul 14 18:37:31.150460 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.150237 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" event={"ID":"8c95bd58-148f-4245-b5d2-6d6b72893c8a","Type":"ContainerDied","Data":"58d909b7f51556c01d2a8badf64a57758a16ecc0ddaed00ed02fc7f8d0f1227a"} Jul 14 18:37:31.150460 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.150295 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf" event={"ID":"8c95bd58-148f-4245-b5d2-6d6b72893c8a","Type":"ContainerDied","Data":"94c41da9fb4f0d4407fe8ca8a71df96ad33c7ba3209d90cfd5a67b2ec40c05f3"} Jul 14 18:37:31.150460 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.150322 2565 scope.go:117] "RemoveContainer" containerID="58d909b7f51556c01d2a8badf64a57758a16ecc0ddaed00ed02fc7f8d0f1227a" Jul 14 18:37:31.159876 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.159857 2565 scope.go:117] "RemoveContainer" containerID="91e0d73ba2ff83c014391db0021cd6c1f4ba55d27454293a8433107be3266991" Jul 14 18:37:31.168719 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.168699 2565 scope.go:117] "RemoveContainer" containerID="58d909b7f51556c01d2a8badf64a57758a16ecc0ddaed00ed02fc7f8d0f1227a" Jul 14 18:37:31.169021 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:37:31.169003 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58d909b7f51556c01d2a8badf64a57758a16ecc0ddaed00ed02fc7f8d0f1227a\": container with ID starting with 58d909b7f51556c01d2a8badf64a57758a16ecc0ddaed00ed02fc7f8d0f1227a not found: ID does not exist" containerID="58d909b7f51556c01d2a8badf64a57758a16ecc0ddaed00ed02fc7f8d0f1227a" Jul 14 18:37:31.169063 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.169031 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58d909b7f51556c01d2a8badf64a57758a16ecc0ddaed00ed02fc7f8d0f1227a"} err="failed to get container status \"58d909b7f51556c01d2a8badf64a57758a16ecc0ddaed00ed02fc7f8d0f1227a\": rpc error: code = NotFound desc = could not find container \"58d909b7f51556c01d2a8badf64a57758a16ecc0ddaed00ed02fc7f8d0f1227a\": container with ID starting with 58d909b7f51556c01d2a8badf64a57758a16ecc0ddaed00ed02fc7f8d0f1227a not found: ID does not exist" Jul 14 18:37:31.169063 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.169049 2565 scope.go:117] "RemoveContainer" containerID="91e0d73ba2ff83c014391db0021cd6c1f4ba55d27454293a8433107be3266991" Jul 14 18:37:31.169328 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:37:31.169310 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91e0d73ba2ff83c014391db0021cd6c1f4ba55d27454293a8433107be3266991\": container with ID starting with 91e0d73ba2ff83c014391db0021cd6c1f4ba55d27454293a8433107be3266991 not found: ID does not exist" containerID="91e0d73ba2ff83c014391db0021cd6c1f4ba55d27454293a8433107be3266991" Jul 14 18:37:31.169381 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.169333 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e0d73ba2ff83c014391db0021cd6c1f4ba55d27454293a8433107be3266991"} err="failed to get container status \"91e0d73ba2ff83c014391db0021cd6c1f4ba55d27454293a8433107be3266991\": rpc error: code = NotFound desc = could not find container \"91e0d73ba2ff83c014391db0021cd6c1f4ba55d27454293a8433107be3266991\": container with ID starting with 91e0d73ba2ff83c014391db0021cd6c1f4ba55d27454293a8433107be3266991 not found: ID does not exist" Jul 14 18:37:31.175127 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.175082 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf"] Jul 14 18:37:31.180155 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.180128 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1808df45-openshift-default-665d5ff65f-7pbtf"] Jul 14 18:37:31.186543 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.186516 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c95bd58-148f-4245-b5d2-6d6b72893c8a-istio-podinfo\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:37:31.186543 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.186542 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c95bd58-148f-4245-b5d2-6d6b72893c8a-istio-token\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:37:31.186709 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.186552 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwgbp\" (UniqueName: \"kubernetes.io/projected/8c95bd58-148f-4245-b5d2-6d6b72893c8a-kube-api-access-bwgbp\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:37:31.186709 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.186561 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c95bd58-148f-4245-b5d2-6d6b72893c8a-istio-envoy\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:37:31.186709 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.186570 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c95bd58-148f-4245-b5d2-6d6b72893c8a-istio-data\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:37:31.186709 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.186579 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c95bd58-148f-4245-b5d2-6d6b72893c8a-workload-certs\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:37:31.186709 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.186587 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c95bd58-148f-4245-b5d2-6d6b72893c8a-istiod-ca-cert\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:37:31.186709 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.186594 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c95bd58-148f-4245-b5d2-6d6b72893c8a-workload-socket\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:37:31.674798 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.674747 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-qg4g2"] Jul 14 18:37:31.675224 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.675200 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b965b04-799c-4da6-96e2-8c5de94a548e" containerName="revoke-keys" Jul 14 18:37:31.675224 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.675219 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b965b04-799c-4da6-96e2-8c5de94a548e" containerName="revoke-keys" Jul 14 18:37:31.675376 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.675238 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b965b04-799c-4da6-96e2-8c5de94a548e" containerName="revoke-keys" Jul 14 18:37:31.675376 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.675244 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b965b04-799c-4da6-96e2-8c5de94a548e" containerName="revoke-keys" Jul 14 18:37:31.675376 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.675251 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c95bd58-148f-4245-b5d2-6d6b72893c8a" containerName="istio-proxy" Jul 14 18:37:31.675376 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.675256 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c95bd58-148f-4245-b5d2-6d6b72893c8a" containerName="istio-proxy" Jul 14 18:37:31.675376 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.675322 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b965b04-799c-4da6-96e2-8c5de94a548e" containerName="revoke-keys" Jul 14 18:37:31.675376 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.675330 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b965b04-799c-4da6-96e2-8c5de94a548e" containerName="revoke-keys" Jul 14 18:37:31.675376 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.675338 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c95bd58-148f-4245-b5d2-6d6b72893c8a" containerName="istio-proxy" Jul 14 18:37:31.675376 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.675344 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c95bd58-148f-4245-b5d2-6d6b72893c8a" containerName="istio-proxy" Jul 14 18:37:31.679615 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.679593 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-qg4g2" Jul 14 18:37:31.682576 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.682555 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 18:37:31.682678 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.682572 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9jplk\"" Jul 14 18:37:31.692624 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.692593 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c5c0f1-509a-44df-a381-4d5317de1797-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e2e80b9d-qg4g2\" (UID: \"e4c5c0f1-509a-44df-a381-4d5317de1797\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-qg4g2" Jul 14 18:37:31.692812 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.692785 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c95bd58-148f-4245-b5d2-6d6b72893c8a" path="/var/lib/kubelet/pods/8c95bd58-148f-4245-b5d2-6d6b72893c8a/volumes" Jul 14 18:37:31.693221 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.693196 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-qg4g2"] Jul 14 18:37:31.794039 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.794001 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c5c0f1-509a-44df-a381-4d5317de1797-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e2e80b9d-qg4g2\" (UID: \"e4c5c0f1-509a-44df-a381-4d5317de1797\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-qg4g2" Jul 14 18:37:31.794842 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.794817 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c5c0f1-509a-44df-a381-4d5317de1797-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e2e80b9d-qg4g2\" (UID: \"e4c5c0f1-509a-44df-a381-4d5317de1797\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-qg4g2" Jul 14 18:37:31.993410 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:31.993317 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-qg4g2" Jul 14 18:37:32.139006 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:32.138980 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-qg4g2"] Jul 14 18:37:32.141440 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:37:32.141409 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c5c0f1_509a_44df_a381_4d5317de1797.slice/crio-d1fa1874d4b81b1ec1fd539acde2c39c71f54e786cb4271158b2316e6ff6feee WatchSource:0}: Error finding container d1fa1874d4b81b1ec1fd539acde2c39c71f54e786cb4271158b2316e6ff6feee: Status 404 returned error can't find the container with id d1fa1874d4b81b1ec1fd539acde2c39c71f54e786cb4271158b2316e6ff6feee Jul 14 18:37:32.157919 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:32.157890 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-qg4g2" event={"ID":"e4c5c0f1-509a-44df-a381-4d5317de1797","Type":"ContainerStarted","Data":"d1fa1874d4b81b1ec1fd539acde2c39c71f54e786cb4271158b2316e6ff6feee"} Jul 14 18:37:33.165479 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:33.165387 2565 generic.go:356] "Generic (PLEG): container finished" podID="e4c5c0f1-509a-44df-a381-4d5317de1797" containerID="78c1ad8ccf9cde3fc52ab57455e9f0bbd6a5c000612cddaaa7b339d582ec0f9e" exitCode=7 Jul 14 18:37:33.165861 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:33.165486 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-qg4g2" event={"ID":"e4c5c0f1-509a-44df-a381-4d5317de1797","Type":"ContainerDied","Data":"78c1ad8ccf9cde3fc52ab57455e9f0bbd6a5c000612cddaaa7b339d582ec0f9e"} Jul 14 18:37:33.165861 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:33.165681 2565 scope.go:117] "RemoveContainer" containerID="78c1ad8ccf9cde3fc52ab57455e9f0bbd6a5c000612cddaaa7b339d582ec0f9e" Jul 14 18:37:34.173010 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:34.172909 2565 generic.go:356] "Generic (PLEG): container finished" podID="e4c5c0f1-509a-44df-a381-4d5317de1797" containerID="6fb196c89fa242876fb4122da319ac4494c72076e98c241aab27914932d1c48a" exitCode=7 Jul 14 18:37:34.173477 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:34.173003 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-qg4g2" event={"ID":"e4c5c0f1-509a-44df-a381-4d5317de1797","Type":"ContainerDied","Data":"6fb196c89fa242876fb4122da319ac4494c72076e98c241aab27914932d1c48a"} Jul 14 18:37:34.173477 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:34.173076 2565 scope.go:117] "RemoveContainer" containerID="78c1ad8ccf9cde3fc52ab57455e9f0bbd6a5c000612cddaaa7b339d582ec0f9e" Jul 14 18:37:34.173477 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:34.173200 2565 scope.go:117] "RemoveContainer" containerID="6fb196c89fa242876fb4122da319ac4494c72076e98c241aab27914932d1c48a" Jul 14 18:37:34.173477 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:37:34.173409 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-e2e80b9d-qg4g2_odh-ai-gateway-infra(e4c5c0f1-509a-44df-a381-4d5317de1797)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-qg4g2" podUID="e4c5c0f1-509a-44df-a381-4d5317de1797" Jul 14 18:37:35.180060 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:35.180030 2565 scope.go:117] "RemoveContainer" containerID="6fb196c89fa242876fb4122da319ac4494c72076e98c241aab27914932d1c48a" Jul 14 18:37:35.180452 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:37:35.180276 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-e2e80b9d-qg4g2_odh-ai-gateway-infra(e4c5c0f1-509a-44df-a381-4d5317de1797)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-qg4g2" podUID="e4c5c0f1-509a-44df-a381-4d5317de1797" Jul 14 18:37:35.352442 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:35.352408 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:37:49.684758 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:49.684726 2565 scope.go:117] "RemoveContainer" containerID="6fb196c89fa242876fb4122da319ac4494c72076e98c241aab27914932d1c48a" Jul 14 18:37:50.254717 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:50.254680 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-qg4g2" event={"ID":"e4c5c0f1-509a-44df-a381-4d5317de1797","Type":"ContainerStarted","Data":"98b8266fb7af91ef4a5e0f308f511a2dc7519097e7d5db7ddfebe6b3ae445795"} Jul 14 18:37:50.274338 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:50.274281 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-qg4g2" podStartSLOduration=19.274265119 podStartE2EDuration="19.274265119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:37:50.272407619 +0000 UTC m=+1675.109750878" watchObservedRunningTime="2026-07-14 18:37:50.274265119 +0000 UTC m=+1675.111608425" Jul 14 18:37:51.260919 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:51.260886 2565 generic.go:356] "Generic (PLEG): container finished" podID="e4c5c0f1-509a-44df-a381-4d5317de1797" containerID="98b8266fb7af91ef4a5e0f308f511a2dc7519097e7d5db7ddfebe6b3ae445795" exitCode=0 Jul 14 18:37:51.261402 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:51.260967 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-qg4g2" event={"ID":"e4c5c0f1-509a-44df-a381-4d5317de1797","Type":"ContainerDied","Data":"98b8266fb7af91ef4a5e0f308f511a2dc7519097e7d5db7ddfebe6b3ae445795"} Jul 14 18:37:51.261402 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:51.261056 2565 scope.go:117] "RemoveContainer" containerID="6fb196c89fa242876fb4122da319ac4494c72076e98c241aab27914932d1c48a" Jul 14 18:37:51.289243 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:51.289214 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-qg4g2"] Jul 14 18:37:52.406241 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:52.406211 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-qg4g2" Jul 14 18:37:52.479250 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:52.479202 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c5c0f1-509a-44df-a381-4d5317de1797-maas-api-service-ca\") pod \"e4c5c0f1-509a-44df-a381-4d5317de1797\" (UID: \"e4c5c0f1-509a-44df-a381-4d5317de1797\") " Jul 14 18:37:52.479574 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:52.479551 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c5c0f1-509a-44df-a381-4d5317de1797-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e4c5c0f1-509a-44df-a381-4d5317de1797" (UID: "e4c5c0f1-509a-44df-a381-4d5317de1797"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:37:52.580219 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:52.580117 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c5c0f1-509a-44df-a381-4d5317de1797-maas-api-service-ca\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:37:53.273460 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:53.273432 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-qg4g2" Jul 14 18:37:53.273638 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:53.273471 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-qg4g2" event={"ID":"e4c5c0f1-509a-44df-a381-4d5317de1797","Type":"ContainerDied","Data":"d1fa1874d4b81b1ec1fd539acde2c39c71f54e786cb4271158b2316e6ff6feee"} Jul 14 18:37:53.273638 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:53.273520 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1fa1874d4b81b1ec1fd539acde2c39c71f54e786cb4271158b2316e6ff6feee" Jul 14 18:37:53.306297 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:53.306253 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-qg4g2"] Jul 14 18:37:53.309667 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:53.309639 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-qg4g2"] Jul 14 18:37:53.689535 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:37:53.689450 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4c5c0f1-509a-44df-a381-4d5317de1797" path="/var/lib/kubelet/pods/e4c5c0f1-509a-44df-a381-4d5317de1797/volumes" Jul 14 18:38:31.741947 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:38:31.741911 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-d8jfl"] Jul 14 18:38:31.742470 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:38:31.742308 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4c5c0f1-509a-44df-a381-4d5317de1797" containerName="revoke-keys" Jul 14 18:38:31.742470 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:38:31.742325 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4c5c0f1-509a-44df-a381-4d5317de1797" containerName="revoke-keys" Jul 14 18:38:31.742470 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:38:31.742338 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4c5c0f1-509a-44df-a381-4d5317de1797" containerName="revoke-keys" Jul 14 18:38:31.742470 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:38:31.742343 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4c5c0f1-509a-44df-a381-4d5317de1797" containerName="revoke-keys" Jul 14 18:38:31.742470 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:38:31.742353 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4c5c0f1-509a-44df-a381-4d5317de1797" containerName="revoke-keys" Jul 14 18:38:31.742470 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:38:31.742359 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4c5c0f1-509a-44df-a381-4d5317de1797" containerName="revoke-keys" Jul 14 18:38:31.742470 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:38:31.742376 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c95bd58-148f-4245-b5d2-6d6b72893c8a" containerName="istio-proxy" Jul 14 18:38:31.742470 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:38:31.742381 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c95bd58-148f-4245-b5d2-6d6b72893c8a" containerName="istio-proxy" Jul 14 18:38:31.742470 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:38:31.742443 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4c5c0f1-509a-44df-a381-4d5317de1797" containerName="revoke-keys" Jul 14 18:38:31.742470 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:38:31.742454 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4c5c0f1-509a-44df-a381-4d5317de1797" containerName="revoke-keys" Jul 14 18:38:31.745472 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:38:31.745454 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-d8jfl" Jul 14 18:38:31.748003 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:38:31.747977 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 18:38:31.748149 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:38:31.748124 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9jplk\"" Jul 14 18:38:31.756044 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:38:31.756013 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-d8jfl"] Jul 14 18:38:31.815559 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:38:31.815522 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be5b1dcb-aff2-4483-b651-6d77460e946e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e2e80b9d-d8jfl\" (UID: \"be5b1dcb-aff2-4483-b651-6d77460e946e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-d8jfl" Jul 14 18:38:31.916759 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:38:31.916706 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be5b1dcb-aff2-4483-b651-6d77460e946e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e2e80b9d-d8jfl\" (UID: \"be5b1dcb-aff2-4483-b651-6d77460e946e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-d8jfl" Jul 14 18:38:31.917469 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:38:31.917447 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be5b1dcb-aff2-4483-b651-6d77460e946e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e2e80b9d-d8jfl\" (UID: \"be5b1dcb-aff2-4483-b651-6d77460e946e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-d8jfl" Jul 14 18:38:32.056462 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:38:32.056420 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-d8jfl" Jul 14 18:38:32.200531 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:38:32.200504 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-d8jfl"] Jul 14 18:38:32.202912 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:38:32.202883 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe5b1dcb_aff2_4483_b651_6d77460e946e.slice/crio-b6d54d198d306f49bf8e6c68afab2f857207a19da412350a02b4dd49ac3d8e18 WatchSource:0}: Error finding container b6d54d198d306f49bf8e6c68afab2f857207a19da412350a02b4dd49ac3d8e18: Status 404 returned error can't find the container with id b6d54d198d306f49bf8e6c68afab2f857207a19da412350a02b4dd49ac3d8e18 Jul 14 18:38:32.460778 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:38:32.460743 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-d8jfl" event={"ID":"be5b1dcb-aff2-4483-b651-6d77460e946e","Type":"ContainerStarted","Data":"b6d54d198d306f49bf8e6c68afab2f857207a19da412350a02b4dd49ac3d8e18"} Jul 14 18:38:33.466401 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:38:33.466359 2565 generic.go:356] "Generic (PLEG): container finished" podID="be5b1dcb-aff2-4483-b651-6d77460e946e" containerID="32cd0bb6149dad238f885032307f8410c3c6c2ca84467efd8f50788f40beb66a" exitCode=0 Jul 14 18:38:33.466401 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:38:33.466407 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-d8jfl" event={"ID":"be5b1dcb-aff2-4483-b651-6d77460e946e","Type":"ContainerDied","Data":"32cd0bb6149dad238f885032307f8410c3c6c2ca84467efd8f50788f40beb66a"} Jul 14 18:38:34.609728 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:38:34.609699 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-d8jfl" Jul 14 18:38:34.641992 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:38:34.641963 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be5b1dcb-aff2-4483-b651-6d77460e946e-maas-api-service-ca\") pod \"be5b1dcb-aff2-4483-b651-6d77460e946e\" (UID: \"be5b1dcb-aff2-4483-b651-6d77460e946e\") " Jul 14 18:38:34.642662 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:38:34.642632 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be5b1dcb-aff2-4483-b651-6d77460e946e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "be5b1dcb-aff2-4483-b651-6d77460e946e" (UID: "be5b1dcb-aff2-4483-b651-6d77460e946e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:38:34.743410 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:38:34.743306 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be5b1dcb-aff2-4483-b651-6d77460e946e-maas-api-service-ca\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:38:35.478313 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:38:35.478277 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-d8jfl" event={"ID":"be5b1dcb-aff2-4483-b651-6d77460e946e","Type":"ContainerDied","Data":"b6d54d198d306f49bf8e6c68afab2f857207a19da412350a02b4dd49ac3d8e18"} Jul 14 18:38:35.478313 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:38:35.478305 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2e80b9d-d8jfl" Jul 14 18:38:35.478551 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:38:35.478335 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6d54d198d306f49bf8e6c68afab2f857207a19da412350a02b4dd49ac3d8e18" Jul 14 18:38:45.152763 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:38:45.152724 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:39:01.157905 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:39:01.157866 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:39:13.156766 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:39:13.156728 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:39:55.727906 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:39:55.727870 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wqpt2_4525dcda-ead6-4800-8d17-58812c289010/console-operator/1.log" Jul 14 18:39:55.728432 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:39:55.727967 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wqpt2_4525dcda-ead6-4800-8d17-58812c289010/console-operator/1.log" Jul 14 18:39:55.731158 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:39:55.731137 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f99bz_5f0ecb09-5c7f-4709-9b4a-1a6a03580822/ovn-acl-logging/0.log" Jul 14 18:39:55.731280 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:39:55.731142 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f99bz_5f0ecb09-5c7f-4709-9b4a-1a6a03580822/ovn-acl-logging/0.log" Jul 14 18:40:22.652598 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:22.652558 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:40:30.052911 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:30.052876 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:40:37.856530 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:37.856492 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg"] Jul 14 18:40:37.856986 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:37.856853 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be5b1dcb-aff2-4483-b651-6d77460e946e" containerName="revoke-keys" Jul 14 18:40:37.856986 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:37.856866 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="be5b1dcb-aff2-4483-b651-6d77460e946e" containerName="revoke-keys" Jul 14 18:40:37.856986 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:37.856930 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="be5b1dcb-aff2-4483-b651-6d77460e946e" containerName="revoke-keys" Jul 14 18:40:37.856986 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:37.856939 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4c5c0f1-509a-44df-a381-4d5317de1797" containerName="revoke-keys" Jul 14 18:40:37.859981 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:37.859964 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:37.862605 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:37.862582 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-467eca84-openshift-default-dockercfg-twkzz\"" Jul 14 18:40:37.878369 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:37.878333 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg"] Jul 14 18:40:37.912241 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:37.912187 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d223fbb6-b448-4f43-8a31-f21fad05166f-istiod-ca-cert\") pod \"e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:37.912458 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:37.912297 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d223fbb6-b448-4f43-8a31-f21fad05166f-istio-podinfo\") pod \"e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:37.912458 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:37.912342 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d223fbb6-b448-4f43-8a31-f21fad05166f-istio-token\") pod \"e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:37.912458 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:37.912383 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d223fbb6-b448-4f43-8a31-f21fad05166f-istio-data\") pod \"e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:37.912624 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:37.912457 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d223fbb6-b448-4f43-8a31-f21fad05166f-workload-certs\") pod \"e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:37.912624 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:37.912511 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d223fbb6-b448-4f43-8a31-f21fad05166f-credential-socket\") pod \"e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:37.912624 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:37.912534 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d223fbb6-b448-4f43-8a31-f21fad05166f-istio-envoy\") pod \"e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:37.912624 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:37.912596 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d223fbb6-b448-4f43-8a31-f21fad05166f-workload-socket\") pod \"e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:37.912767 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:37.912632 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8g85\" (UniqueName: \"kubernetes.io/projected/d223fbb6-b448-4f43-8a31-f21fad05166f-kube-api-access-z8g85\") pod \"e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:38.014077 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:38.014043 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d223fbb6-b448-4f43-8a31-f21fad05166f-credential-socket\") pod \"e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:38.014077 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:38.014079 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d223fbb6-b448-4f43-8a31-f21fad05166f-istio-envoy\") pod \"e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:38.014357 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:38.014131 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d223fbb6-b448-4f43-8a31-f21fad05166f-workload-socket\") pod \"e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:38.014357 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:38.014153 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8g85\" (UniqueName: \"kubernetes.io/projected/d223fbb6-b448-4f43-8a31-f21fad05166f-kube-api-access-z8g85\") pod \"e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:38.014357 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:38.014191 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d223fbb6-b448-4f43-8a31-f21fad05166f-istiod-ca-cert\") pod \"e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:38.014357 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:38.014213 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d223fbb6-b448-4f43-8a31-f21fad05166f-istio-podinfo\") pod \"e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:38.014357 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:38.014227 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d223fbb6-b448-4f43-8a31-f21fad05166f-istio-token\") pod \"e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:38.014357 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:38.014255 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d223fbb6-b448-4f43-8a31-f21fad05166f-istio-data\") pod \"e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:38.014357 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:38.014312 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d223fbb6-b448-4f43-8a31-f21fad05166f-workload-certs\") pod \"e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:38.014709 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:38.014546 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d223fbb6-b448-4f43-8a31-f21fad05166f-credential-socket\") pod \"e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:38.014709 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:38.014616 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d223fbb6-b448-4f43-8a31-f21fad05166f-workload-socket\") pod \"e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:38.014709 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:38.014649 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d223fbb6-b448-4f43-8a31-f21fad05166f-workload-certs\") pod \"e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:38.014849 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:38.014831 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d223fbb6-b448-4f43-8a31-f21fad05166f-istio-data\") pod \"e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:38.015609 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:38.015579 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d223fbb6-b448-4f43-8a31-f21fad05166f-istiod-ca-cert\") pod \"e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:38.016805 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:38.016781 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d223fbb6-b448-4f43-8a31-f21fad05166f-istio-envoy\") pod \"e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:38.017089 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:38.017073 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d223fbb6-b448-4f43-8a31-f21fad05166f-istio-podinfo\") pod \"e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:38.023476 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:38.023443 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d223fbb6-b448-4f43-8a31-f21fad05166f-istio-token\") pod \"e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:38.023755 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:38.023731 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8g85\" (UniqueName: \"kubernetes.io/projected/d223fbb6-b448-4f43-8a31-f21fad05166f-kube-api-access-z8g85\") pod \"e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:38.056830 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:38.056769 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:40:38.170434 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:38.170343 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:38.322279 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:38.322245 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg"] Jul 14 18:40:38.324181 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:40:38.324152 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd223fbb6_b448_4f43_8a31_f21fad05166f.slice/crio-79f3cecc49c4cf263dbc898c2885a5a1c819c7be776adab20aed3d627b92cbbf WatchSource:0}: Error finding container 79f3cecc49c4cf263dbc898c2885a5a1c819c7be776adab20aed3d627b92cbbf: Status 404 returned error can't find the container with id 79f3cecc49c4cf263dbc898c2885a5a1c819c7be776adab20aed3d627b92cbbf Jul 14 18:40:38.326357 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:38.326323 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:40:38.326453 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:38.326388 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:40:38.326453 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:38.326419 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:40:39.068568 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:39.068529 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" event={"ID":"d223fbb6-b448-4f43-8a31-f21fad05166f","Type":"ContainerStarted","Data":"0a8880f27ab5822b2f6c34d5ff5287a535b054d395d487d8180522aa443f3ec4"} Jul 14 18:40:39.069003 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:39.068585 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" event={"ID":"d223fbb6-b448-4f43-8a31-f21fad05166f","Type":"ContainerStarted","Data":"79f3cecc49c4cf263dbc898c2885a5a1c819c7be776adab20aed3d627b92cbbf"} Jul 14 18:40:39.094458 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:39.094402 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" podStartSLOduration=2.094383692 podStartE2EDuration="2.094383692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:40:39.092394995 +0000 UTC m=+1843.929738255" watchObservedRunningTime="2026-07-14 18:40:39.094383692 +0000 UTC m=+1843.931726950" Jul 14 18:40:39.170565 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:39.170526 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:39.175609 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:39.175582 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:40.074089 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:40.074056 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:40.075133 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:40.075090 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:47.957375 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:47.957342 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:40:53.885033 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:53.884990 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-467eca84-cvmcl"] Jul 14 18:40:53.889691 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:53.889666 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-467eca84-cvmcl" Jul 14 18:40:53.896126 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:53.896086 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 18:40:53.896536 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:53.896515 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9jplk\"" Jul 14 18:40:53.911644 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:53.911613 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-467eca84-cvmcl"] Jul 14 18:40:54.062265 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:54.062230 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c6396bc-0d5a-4819-9d4d-fbdb43cf740d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-467eca84-cvmcl\" (UID: \"9c6396bc-0d5a-4819-9d4d-fbdb43cf740d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-467eca84-cvmcl" Jul 14 18:40:54.163305 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:54.163215 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c6396bc-0d5a-4819-9d4d-fbdb43cf740d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-467eca84-cvmcl\" (UID: \"9c6396bc-0d5a-4819-9d4d-fbdb43cf740d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-467eca84-cvmcl" Jul 14 18:40:54.164115 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:54.164075 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c6396bc-0d5a-4819-9d4d-fbdb43cf740d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-467eca84-cvmcl\" (UID: \"9c6396bc-0d5a-4819-9d4d-fbdb43cf740d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-467eca84-cvmcl" Jul 14 18:40:54.202859 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:54.202815 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-467eca84-cvmcl" Jul 14 18:40:54.351957 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:54.351926 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-467eca84-cvmcl"] Jul 14 18:40:54.354792 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:40:54.354761 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c6396bc_0d5a_4819_9d4d_fbdb43cf740d.slice/crio-c9561e54e24e39c174f12e3b4f4bb8de7de0012b942f88fc3c8cecb26cba9d31 WatchSource:0}: Error finding container c9561e54e24e39c174f12e3b4f4bb8de7de0012b942f88fc3c8cecb26cba9d31: Status 404 returned error can't find the container with id c9561e54e24e39c174f12e3b4f4bb8de7de0012b942f88fc3c8cecb26cba9d31 Jul 14 18:40:55.154778 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:55.154743 2565 generic.go:356] "Generic (PLEG): container finished" podID="9c6396bc-0d5a-4819-9d4d-fbdb43cf740d" containerID="e3a5d0159aa9666a6e55f7f690e2569409561ad0350736b8646b6270a4312cc8" exitCode=0 Jul 14 18:40:55.155174 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:55.154825 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-467eca84-cvmcl" event={"ID":"9c6396bc-0d5a-4819-9d4d-fbdb43cf740d","Type":"ContainerDied","Data":"e3a5d0159aa9666a6e55f7f690e2569409561ad0350736b8646b6270a4312cc8"} Jul 14 18:40:55.155174 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:55.154892 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-467eca84-cvmcl" event={"ID":"9c6396bc-0d5a-4819-9d4d-fbdb43cf740d","Type":"ContainerStarted","Data":"c9561e54e24e39c174f12e3b4f4bb8de7de0012b942f88fc3c8cecb26cba9d31"} Jul 14 18:40:55.655530 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:55.655266 2565 patch_prober.go:28] interesting pod/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": read tcp 10.132.0.2:58512->10.132.0.43:15021: read: connection reset by peer" start-of-body= Jul 14 18:40:55.655530 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:55.655338 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" podUID="d223fbb6-b448-4f43-8a31-f21fad05166f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": read tcp 10.132.0.2:58512->10.132.0.43:15021: read: connection reset by peer" Jul 14 18:40:56.160956 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:56.160927 2565 generic.go:356] "Generic (PLEG): container finished" podID="d223fbb6-b448-4f43-8a31-f21fad05166f" containerID="0a8880f27ab5822b2f6c34d5ff5287a535b054d395d487d8180522aa443f3ec4" exitCode=0 Jul 14 18:40:56.161415 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:56.161015 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" event={"ID":"d223fbb6-b448-4f43-8a31-f21fad05166f","Type":"ContainerDied","Data":"0a8880f27ab5822b2f6c34d5ff5287a535b054d395d487d8180522aa443f3ec4"} Jul 14 18:40:56.161773 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:56.161517 2565 scope.go:117] "RemoveContainer" containerID="0a8880f27ab5822b2f6c34d5ff5287a535b054d395d487d8180522aa443f3ec4" Jul 14 18:40:56.162699 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:56.162661 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:40:56.162815 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:56.162774 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:40:56.162863 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:56.162825 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:40:56.340553 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:56.340527 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-467eca84-cvmcl" Jul 14 18:40:56.486781 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:56.486682 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c6396bc-0d5a-4819-9d4d-fbdb43cf740d-maas-api-service-ca\") pod \"9c6396bc-0d5a-4819-9d4d-fbdb43cf740d\" (UID: \"9c6396bc-0d5a-4819-9d4d-fbdb43cf740d\") " Jul 14 18:40:56.487015 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:56.486993 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c6396bc-0d5a-4819-9d4d-fbdb43cf740d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9c6396bc-0d5a-4819-9d4d-fbdb43cf740d" (UID: "9c6396bc-0d5a-4819-9d4d-fbdb43cf740d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:40:56.487133 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:56.487079 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c6396bc-0d5a-4819-9d4d-fbdb43cf740d-maas-api-service-ca\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:40:57.167629 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:57.167592 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-467eca84-cvmcl" Jul 14 18:40:57.167629 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:57.167625 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-467eca84-cvmcl" event={"ID":"9c6396bc-0d5a-4819-9d4d-fbdb43cf740d","Type":"ContainerDied","Data":"c9561e54e24e39c174f12e3b4f4bb8de7de0012b942f88fc3c8cecb26cba9d31"} Jul 14 18:40:57.168214 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:57.167675 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9561e54e24e39c174f12e3b4f4bb8de7de0012b942f88fc3c8cecb26cba9d31" Jul 14 18:40:57.169739 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:57.169717 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" event={"ID":"d223fbb6-b448-4f43-8a31-f21fad05166f","Type":"ContainerStarted","Data":"a499a69c29ef7aa75fd1ca089d95146d56891b8443d4fc9402275efb5b403f63"} Jul 14 18:40:57.170693 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:57.170678 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:40:57.170894 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:57.170874 2565 patch_prober.go:28] interesting pod/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 14 18:40:57.170975 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:57.170925 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" podUID="d223fbb6-b448-4f43-8a31-f21fad05166f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 14 18:40:58.170797 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:58.170747 2565 patch_prober.go:28] interesting pod/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 14 18:40:58.171203 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:58.170857 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" podUID="d223fbb6-b448-4f43-8a31-f21fad05166f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 14 18:40:59.171382 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:59.171346 2565 patch_prober.go:28] interesting pod/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 14 18:40:59.171778 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:40:59.171400 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" podUID="d223fbb6-b448-4f43-8a31-f21fad05166f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 14 18:41:00.170677 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:00.170638 2565 patch_prober.go:28] interesting pod/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 14 18:41:00.170921 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:00.170695 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" podUID="d223fbb6-b448-4f43-8a31-f21fad05166f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 14 18:41:01.174767 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:01.174735 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:41:01.175262 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:01.175009 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:41:01.175658 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:01.175642 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:41:07.446548 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:07.446513 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:41:19.753655 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:19.753568 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg"] Jul 14 18:41:19.754621 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:19.753843 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" podUID="d223fbb6-b448-4f43-8a31-f21fad05166f" containerName="istio-proxy" containerID="cri-o://a499a69c29ef7aa75fd1ca089d95146d56891b8443d4fc9402275efb5b403f63" gracePeriod=30 Jul 14 18:41:25.012856 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.012804 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:41:25.142904 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.142875 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d223fbb6-b448-4f43-8a31-f21fad05166f-istiod-ca-cert\") pod \"d223fbb6-b448-4f43-8a31-f21fad05166f\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " Jul 14 18:41:25.143080 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.142921 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d223fbb6-b448-4f43-8a31-f21fad05166f-workload-certs\") pod \"d223fbb6-b448-4f43-8a31-f21fad05166f\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " Jul 14 18:41:25.143080 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.142953 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d223fbb6-b448-4f43-8a31-f21fad05166f-workload-socket\") pod \"d223fbb6-b448-4f43-8a31-f21fad05166f\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " Jul 14 18:41:25.143080 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.142973 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d223fbb6-b448-4f43-8a31-f21fad05166f-istio-envoy\") pod \"d223fbb6-b448-4f43-8a31-f21fad05166f\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " Jul 14 18:41:25.143080 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.143014 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8g85\" (UniqueName: \"kubernetes.io/projected/d223fbb6-b448-4f43-8a31-f21fad05166f-kube-api-access-z8g85\") pod \"d223fbb6-b448-4f43-8a31-f21fad05166f\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " Jul 14 18:41:25.143080 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.143042 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d223fbb6-b448-4f43-8a31-f21fad05166f-istio-podinfo\") pod \"d223fbb6-b448-4f43-8a31-f21fad05166f\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " Jul 14 18:41:25.143080 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.143064 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d223fbb6-b448-4f43-8a31-f21fad05166f-istio-token\") pod \"d223fbb6-b448-4f43-8a31-f21fad05166f\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " Jul 14 18:41:25.143397 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.143137 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d223fbb6-b448-4f43-8a31-f21fad05166f-credential-socket\") pod \"d223fbb6-b448-4f43-8a31-f21fad05166f\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " Jul 14 18:41:25.143397 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.143167 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d223fbb6-b448-4f43-8a31-f21fad05166f-istio-data\") pod \"d223fbb6-b448-4f43-8a31-f21fad05166f\" (UID: \"d223fbb6-b448-4f43-8a31-f21fad05166f\") " Jul 14 18:41:25.143397 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.143337 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d223fbb6-b448-4f43-8a31-f21fad05166f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d223fbb6-b448-4f43-8a31-f21fad05166f" (UID: "d223fbb6-b448-4f43-8a31-f21fad05166f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:41:25.143397 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.143349 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d223fbb6-b448-4f43-8a31-f21fad05166f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d223fbb6-b448-4f43-8a31-f21fad05166f" (UID: "d223fbb6-b448-4f43-8a31-f21fad05166f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:41:25.143397 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.143358 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d223fbb6-b448-4f43-8a31-f21fad05166f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d223fbb6-b448-4f43-8a31-f21fad05166f" (UID: "d223fbb6-b448-4f43-8a31-f21fad05166f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:41:25.143640 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.143449 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d223fbb6-b448-4f43-8a31-f21fad05166f-istiod-ca-cert\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:41:25.143640 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.143463 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d223fbb6-b448-4f43-8a31-f21fad05166f-workload-certs\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:41:25.143640 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.143502 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d223fbb6-b448-4f43-8a31-f21fad05166f-workload-socket\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:41:25.143640 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.143568 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d223fbb6-b448-4f43-8a31-f21fad05166f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d223fbb6-b448-4f43-8a31-f21fad05166f" (UID: "d223fbb6-b448-4f43-8a31-f21fad05166f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:41:25.144425 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.144399 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d223fbb6-b448-4f43-8a31-f21fad05166f-istio-data" (OuterVolumeSpecName: "istio-data") pod "d223fbb6-b448-4f43-8a31-f21fad05166f" (UID: "d223fbb6-b448-4f43-8a31-f21fad05166f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:41:25.145426 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.145394 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d223fbb6-b448-4f43-8a31-f21fad05166f-kube-api-access-z8g85" (OuterVolumeSpecName: "kube-api-access-z8g85") pod "d223fbb6-b448-4f43-8a31-f21fad05166f" (UID: "d223fbb6-b448-4f43-8a31-f21fad05166f"). InnerVolumeSpecName "kube-api-access-z8g85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:41:25.145582 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.145562 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d223fbb6-b448-4f43-8a31-f21fad05166f-istio-token" (OuterVolumeSpecName: "istio-token") pod "d223fbb6-b448-4f43-8a31-f21fad05166f" (UID: "d223fbb6-b448-4f43-8a31-f21fad05166f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:41:25.146032 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.146003 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d223fbb6-b448-4f43-8a31-f21fad05166f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d223fbb6-b448-4f43-8a31-f21fad05166f" (UID: "d223fbb6-b448-4f43-8a31-f21fad05166f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 18:41:25.146032 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.145999 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d223fbb6-b448-4f43-8a31-f21fad05166f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d223fbb6-b448-4f43-8a31-f21fad05166f" (UID: "d223fbb6-b448-4f43-8a31-f21fad05166f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:41:25.244566 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.244534 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8g85\" (UniqueName: \"kubernetes.io/projected/d223fbb6-b448-4f43-8a31-f21fad05166f-kube-api-access-z8g85\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:41:25.244566 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.244560 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d223fbb6-b448-4f43-8a31-f21fad05166f-istio-podinfo\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:41:25.244566 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.244570 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d223fbb6-b448-4f43-8a31-f21fad05166f-istio-token\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:41:25.244767 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.244578 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d223fbb6-b448-4f43-8a31-f21fad05166f-credential-socket\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:41:25.244767 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.244587 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d223fbb6-b448-4f43-8a31-f21fad05166f-istio-data\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:41:25.244767 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.244596 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d223fbb6-b448-4f43-8a31-f21fad05166f-istio-envoy\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:41:25.321146 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.321109 2565 generic.go:356] "Generic (PLEG): container finished" podID="d223fbb6-b448-4f43-8a31-f21fad05166f" containerID="a499a69c29ef7aa75fd1ca089d95146d56891b8443d4fc9402275efb5b403f63" exitCode=0 Jul 14 18:41:25.321297 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.321152 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" event={"ID":"d223fbb6-b448-4f43-8a31-f21fad05166f","Type":"ContainerDied","Data":"a499a69c29ef7aa75fd1ca089d95146d56891b8443d4fc9402275efb5b403f63"} Jul 14 18:41:25.321297 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.321162 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" Jul 14 18:41:25.321297 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.321198 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg" event={"ID":"d223fbb6-b448-4f43-8a31-f21fad05166f","Type":"ContainerDied","Data":"79f3cecc49c4cf263dbc898c2885a5a1c819c7be776adab20aed3d627b92cbbf"} Jul 14 18:41:25.321297 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.321224 2565 scope.go:117] "RemoveContainer" containerID="a499a69c29ef7aa75fd1ca089d95146d56891b8443d4fc9402275efb5b403f63" Jul 14 18:41:25.331175 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.331151 2565 scope.go:117] "RemoveContainer" containerID="0a8880f27ab5822b2f6c34d5ff5287a535b054d395d487d8180522aa443f3ec4" Jul 14 18:41:25.339448 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.339427 2565 scope.go:117] "RemoveContainer" containerID="a499a69c29ef7aa75fd1ca089d95146d56891b8443d4fc9402275efb5b403f63" Jul 14 18:41:25.339707 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:41:25.339689 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a499a69c29ef7aa75fd1ca089d95146d56891b8443d4fc9402275efb5b403f63\": container with ID starting with a499a69c29ef7aa75fd1ca089d95146d56891b8443d4fc9402275efb5b403f63 not found: ID does not exist" containerID="a499a69c29ef7aa75fd1ca089d95146d56891b8443d4fc9402275efb5b403f63" Jul 14 18:41:25.339773 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.339714 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a499a69c29ef7aa75fd1ca089d95146d56891b8443d4fc9402275efb5b403f63"} err="failed to get container status \"a499a69c29ef7aa75fd1ca089d95146d56891b8443d4fc9402275efb5b403f63\": rpc error: code = NotFound desc = could not find container \"a499a69c29ef7aa75fd1ca089d95146d56891b8443d4fc9402275efb5b403f63\": container with ID starting with a499a69c29ef7aa75fd1ca089d95146d56891b8443d4fc9402275efb5b403f63 not found: ID does not exist" Jul 14 18:41:25.339773 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.339732 2565 scope.go:117] "RemoveContainer" containerID="0a8880f27ab5822b2f6c34d5ff5287a535b054d395d487d8180522aa443f3ec4" Jul 14 18:41:25.339970 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:41:25.339956 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a8880f27ab5822b2f6c34d5ff5287a535b054d395d487d8180522aa443f3ec4\": container with ID starting with 0a8880f27ab5822b2f6c34d5ff5287a535b054d395d487d8180522aa443f3ec4 not found: ID does not exist" containerID="0a8880f27ab5822b2f6c34d5ff5287a535b054d395d487d8180522aa443f3ec4" Jul 14 18:41:25.340016 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.339973 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a8880f27ab5822b2f6c34d5ff5287a535b054d395d487d8180522aa443f3ec4"} err="failed to get container status \"0a8880f27ab5822b2f6c34d5ff5287a535b054d395d487d8180522aa443f3ec4\": rpc error: code = NotFound desc = could not find container \"0a8880f27ab5822b2f6c34d5ff5287a535b054d395d487d8180522aa443f3ec4\": container with ID starting with 0a8880f27ab5822b2f6c34d5ff5287a535b054d395d487d8180522aa443f3ec4 not found: ID does not exist" Jul 14 18:41:25.345064 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.345037 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg"] Jul 14 18:41:25.349533 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.349510 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-467eca84-openshift-default-ccd8f4fb7-shdgg"] Jul 14 18:41:25.689864 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:25.689783 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d223fbb6-b448-4f43-8a31-f21fad05166f" path="/var/lib/kubelet/pods/d223fbb6-b448-4f43-8a31-f21fad05166f/volumes" Jul 14 18:41:41.146127 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.146059 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5"] Jul 14 18:41:41.146637 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.146615 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d223fbb6-b448-4f43-8a31-f21fad05166f" containerName="istio-proxy" Jul 14 18:41:41.146715 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.146641 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d223fbb6-b448-4f43-8a31-f21fad05166f" containerName="istio-proxy" Jul 14 18:41:41.146715 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.146668 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c6396bc-0d5a-4819-9d4d-fbdb43cf740d" containerName="revoke-keys" Jul 14 18:41:41.146715 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.146677 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c6396bc-0d5a-4819-9d4d-fbdb43cf740d" containerName="revoke-keys" Jul 14 18:41:41.146883 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.146805 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c6396bc-0d5a-4819-9d4d-fbdb43cf740d" containerName="revoke-keys" Jul 14 18:41:41.146883 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.146819 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d223fbb6-b448-4f43-8a31-f21fad05166f" containerName="istio-proxy" Jul 14 18:41:41.149860 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.149839 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:41.152698 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.152679 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c99ca1c1-openshift-default-dockercfg-rxkvs\"" Jul 14 18:41:41.161077 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.161053 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5"] Jul 14 18:41:41.285183 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.285135 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-istio-podinfo\") pod \"e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:41.285365 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.285220 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-workload-certs\") pod \"e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:41.285365 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.285265 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-credential-socket\") pod \"e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:41.285365 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.285289 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-istio-envoy\") pod \"e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:41.285365 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.285327 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc79c\" (UniqueName: \"kubernetes.io/projected/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-kube-api-access-vc79c\") pod \"e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:41.285561 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.285383 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-istio-token\") pod \"e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:41.285561 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.285406 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-istiod-ca-cert\") pod \"e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:41.285561 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.285432 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-workload-socket\") pod \"e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:41.285561 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.285454 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-istio-data\") pod \"e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:41.393517 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.393480 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-istio-token\") pod \"e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:41.393708 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.393659 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-istiod-ca-cert\") pod \"e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:41.393774 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.393708 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-workload-socket\") pod \"e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:41.393774 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.393729 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-istio-data\") pod \"e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:41.393884 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.393825 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-istio-podinfo\") pod \"e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:41.393936 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.393882 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-workload-certs\") pod \"e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:41.393936 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.393920 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-credential-socket\") pod \"e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:41.394042 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.393943 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-istio-envoy\") pod \"e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:41.394042 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.393983 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc79c\" (UniqueName: \"kubernetes.io/projected/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-kube-api-access-vc79c\") pod \"e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:41.394196 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.394161 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-workload-socket\") pod \"e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:41.394421 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.394396 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-istiod-ca-cert\") pod \"e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:41.394570 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.394448 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-credential-socket\") pod \"e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:41.394745 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.394717 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-workload-certs\") pod \"e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:41.394930 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.394908 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-istio-data\") pod \"e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:41.396687 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.396635 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-istio-envoy\") pod \"e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:41.396985 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.396969 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-istio-podinfo\") pod \"e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:41.403174 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.403151 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-istio-token\") pod \"e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:41.403268 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.403173 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc79c\" (UniqueName: \"kubernetes.io/projected/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-kube-api-access-vc79c\") pod \"e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:41.462377 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.462343 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:41.609066 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.609035 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5"] Jul 14 18:41:41.610914 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:41:41.610888 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1bd6c1_4939_4bf6_8152_5fd4e4957e91.slice/crio-ba2037bd16b0d370c711abbf520201be7fe13f61c5de042b8c076264da9ca0ba WatchSource:0}: Error finding container ba2037bd16b0d370c711abbf520201be7fe13f61c5de042b8c076264da9ca0ba: Status 404 returned error can't find the container with id ba2037bd16b0d370c711abbf520201be7fe13f61c5de042b8c076264da9ca0ba Jul 14 18:41:41.613800 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.613758 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:41:41.613898 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.613851 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:41:41.613898 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:41.613892 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:41:42.415915 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:42.415867 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" event={"ID":"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91","Type":"ContainerStarted","Data":"fc20195997d671f86460bdb3676a5ccb972329a51406ba2697462325aa86fdf2"} Jul 14 18:41:42.416368 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:42.415937 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" event={"ID":"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91","Type":"ContainerStarted","Data":"ba2037bd16b0d370c711abbf520201be7fe13f61c5de042b8c076264da9ca0ba"} Jul 14 18:41:42.445563 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:42.445491 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" podStartSLOduration=1.44547052 podStartE2EDuration="1.44547052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:41:42.444570504 +0000 UTC m=+1907.281913761" watchObservedRunningTime="2026-07-14 18:41:42.44547052 +0000 UTC m=+1907.282813777" Jul 14 18:41:42.463155 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:42.463090 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:42.468234 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:42.468211 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:43.420430 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:43.420400 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:43.421465 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:43.421448 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:41:50.047191 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:50.047151 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:41:58.422363 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:58.422327 2565 patch_prober.go:28] interesting pod/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 14 18:41:58.422890 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:58.422392 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" podUID="7c1bd6c1-4939-4bf6-8152-5fd4e4957e91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 14 18:41:58.501608 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:58.501575 2565 generic.go:356] "Generic (PLEG): container finished" podID="7c1bd6c1-4939-4bf6-8152-5fd4e4957e91" containerID="fc20195997d671f86460bdb3676a5ccb972329a51406ba2697462325aa86fdf2" exitCode=0 Jul 14 18:41:58.501817 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:58.501628 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" event={"ID":"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91","Type":"ContainerDied","Data":"fc20195997d671f86460bdb3676a5ccb972329a51406ba2697462325aa86fdf2"} Jul 14 18:41:58.502084 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:58.502066 2565 scope.go:117] "RemoveContainer" containerID="fc20195997d671f86460bdb3676a5ccb972329a51406ba2697462325aa86fdf2" Jul 14 18:41:58.503154 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:58.503125 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:41:58.503248 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:58.503208 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:41:58.503310 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:58.503261 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:41:59.508981 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:41:59.508943 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" event={"ID":"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91","Type":"ContainerStarted","Data":"4c0a1d7f2c857d5919800cf66c40efc3aebb393ffb3d1399b1fb8f4fa33ec621"} Jul 14 18:42:00.463434 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:00.463392 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:42:00.463663 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:00.463639 2565 patch_prober.go:28] interesting pod/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 14 18:42:00.463716 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:00.463700 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" podUID="7c1bd6c1-4939-4bf6-8152-5fd4e4957e91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 14 18:42:00.962084 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:00.962036 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:42:01.463537 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:01.463496 2565 patch_prober.go:28] interesting pod/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 14 18:42:01.463729 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:01.463564 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" podUID="7c1bd6c1-4939-4bf6-8152-5fd4e4957e91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 14 18:42:02.463582 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:02.463544 2565 patch_prober.go:28] interesting pod/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 14 18:42:02.464032 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:02.463601 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" podUID="7c1bd6c1-4939-4bf6-8152-5fd4e4957e91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 14 18:42:03.462938 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:03.462893 2565 patch_prober.go:28] interesting pod/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 14 18:42:03.463190 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:03.462973 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" podUID="7c1bd6c1-4939-4bf6-8152-5fd4e4957e91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 14 18:42:04.467363 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:04.467326 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:42:04.467759 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:04.467626 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:42:04.468338 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:04.468316 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:42:21.254001 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:21.253955 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:42:27.488987 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:27.488947 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99ca1c1-z9h7b"] Jul 14 18:42:27.489641 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:27.489366 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d223fbb6-b448-4f43-8a31-f21fad05166f" containerName="istio-proxy" Jul 14 18:42:27.489641 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:27.489379 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d223fbb6-b448-4f43-8a31-f21fad05166f" containerName="istio-proxy" Jul 14 18:42:27.489641 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:27.489444 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d223fbb6-b448-4f43-8a31-f21fad05166f" containerName="istio-proxy" Jul 14 18:42:27.493669 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:27.493647 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99ca1c1-z9h7b" Jul 14 18:42:27.496592 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:27.496565 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 18:42:27.496764 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:27.496650 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9jplk\"" Jul 14 18:42:27.500894 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:27.500858 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99ca1c1-z9h7b"] Jul 14 18:42:27.531716 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:27.531674 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a9f60c-0c98-4374-a7e1-95f35a4a6c66-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c99ca1c1-z9h7b\" (UID: \"21a9f60c-0c98-4374-a7e1-95f35a4a6c66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99ca1c1-z9h7b" Jul 14 18:42:27.632151 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:27.632087 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a9f60c-0c98-4374-a7e1-95f35a4a6c66-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c99ca1c1-z9h7b\" (UID: \"21a9f60c-0c98-4374-a7e1-95f35a4a6c66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99ca1c1-z9h7b" Jul 14 18:42:27.632801 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:27.632779 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a9f60c-0c98-4374-a7e1-95f35a4a6c66-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c99ca1c1-z9h7b\" (UID: \"21a9f60c-0c98-4374-a7e1-95f35a4a6c66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99ca1c1-z9h7b" Jul 14 18:42:27.805726 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:27.805684 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99ca1c1-z9h7b" Jul 14 18:42:27.946261 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:27.946224 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99ca1c1-z9h7b"] Jul 14 18:42:27.949697 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:42:27.949661 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a9f60c_0c98_4374_a7e1_95f35a4a6c66.slice/crio-1042c64dd9ad437db813476945f92164ec95e5d7301d53c9bc1d93052e7fe24b WatchSource:0}: Error finding container 1042c64dd9ad437db813476945f92164ec95e5d7301d53c9bc1d93052e7fe24b: Status 404 returned error can't find the container with id 1042c64dd9ad437db813476945f92164ec95e5d7301d53c9bc1d93052e7fe24b Jul 14 18:42:28.667080 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:28.667042 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99ca1c1-z9h7b" event={"ID":"21a9f60c-0c98-4374-a7e1-95f35a4a6c66","Type":"ContainerStarted","Data":"82a95fc7f4ddf6684a0b40a8507e6e0cd7f9cecbab7226fcee675cd47db11e7a"} Jul 14 18:42:28.667080 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:28.667112 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99ca1c1-z9h7b" event={"ID":"21a9f60c-0c98-4374-a7e1-95f35a4a6c66","Type":"ContainerStarted","Data":"1042c64dd9ad437db813476945f92164ec95e5d7301d53c9bc1d93052e7fe24b"} Jul 14 18:42:28.689799 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:28.689748 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99ca1c1-z9h7b" podStartSLOduration=1.689732325 podStartE2EDuration="1.689732325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:42:28.689518081 +0000 UTC m=+1953.526861338" watchObservedRunningTime="2026-07-14 18:42:28.689732325 +0000 UTC m=+1953.527075581" Jul 14 18:42:29.673839 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:29.673807 2565 generic.go:356] "Generic (PLEG): container finished" podID="21a9f60c-0c98-4374-a7e1-95f35a4a6c66" containerID="82a95fc7f4ddf6684a0b40a8507e6e0cd7f9cecbab7226fcee675cd47db11e7a" exitCode=0 Jul 14 18:42:29.674260 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:29.673895 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99ca1c1-z9h7b" event={"ID":"21a9f60c-0c98-4374-a7e1-95f35a4a6c66","Type":"ContainerDied","Data":"82a95fc7f4ddf6684a0b40a8507e6e0cd7f9cecbab7226fcee675cd47db11e7a"} Jul 14 18:42:30.818377 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:30.818354 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99ca1c1-z9h7b" Jul 14 18:42:30.858162 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:30.858131 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a9f60c-0c98-4374-a7e1-95f35a4a6c66-maas-api-service-ca\") pod \"21a9f60c-0c98-4374-a7e1-95f35a4a6c66\" (UID: \"21a9f60c-0c98-4374-a7e1-95f35a4a6c66\") " Jul 14 18:42:30.858492 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:30.858469 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21a9f60c-0c98-4374-a7e1-95f35a4a6c66-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "21a9f60c-0c98-4374-a7e1-95f35a4a6c66" (UID: "21a9f60c-0c98-4374-a7e1-95f35a4a6c66"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:42:30.959785 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:30.959681 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a9f60c-0c98-4374-a7e1-95f35a4a6c66-maas-api-service-ca\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:42:31.685210 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:31.685177 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99ca1c1-z9h7b" Jul 14 18:42:31.689838 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:31.689809 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99ca1c1-z9h7b" event={"ID":"21a9f60c-0c98-4374-a7e1-95f35a4a6c66","Type":"ContainerDied","Data":"1042c64dd9ad437db813476945f92164ec95e5d7301d53c9bc1d93052e7fe24b"} Jul 14 18:42:31.689991 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:31.689858 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1042c64dd9ad437db813476945f92164ec95e5d7301d53c9bc1d93052e7fe24b" Jul 14 18:42:41.951868 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:41.951831 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:42:55.811867 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:42:55.811836 2565 scope.go:117] "RemoveContainer" containerID="299bae6d501dd4745a1a6af764b452682f5d0976c5b36594587a528fc5ad0adf" Jul 14 18:43:35.247219 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:35.247185 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5"] Jul 14 18:43:35.247691 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:35.247380 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" podUID="7c1bd6c1-4939-4bf6-8152-5fd4e4957e91" containerName="istio-proxy" containerID="cri-o://4c0a1d7f2c857d5919800cf66c40efc3aebb393ffb3d1399b1fb8f4fa33ec621" gracePeriod=30 Jul 14 18:43:40.510848 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:40.510791 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:43:40.573563 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:40.573532 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vc79c\" (UniqueName: \"kubernetes.io/projected/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-kube-api-access-vc79c\") pod \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " Jul 14 18:43:40.573725 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:40.573568 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-istio-envoy\") pod \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " Jul 14 18:43:40.573725 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:40.573606 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-workload-socket\") pod \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " Jul 14 18:43:40.573725 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:40.573655 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-workload-certs\") pod \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " Jul 14 18:43:40.573725 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:40.573680 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-credential-socket\") pod \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " Jul 14 18:43:40.573725 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:40.573695 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-istio-data\") pod \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " Jul 14 18:43:40.573982 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:40.573730 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-istiod-ca-cert\") pod \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " Jul 14 18:43:40.573982 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:40.573775 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-istio-podinfo\") pod \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " Jul 14 18:43:40.573982 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:40.573797 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-istio-token\") pod \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\" (UID: \"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91\") " Jul 14 18:43:40.574168 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:40.573995 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7c1bd6c1-4939-4bf6-8152-5fd4e4957e91" (UID: "7c1bd6c1-4939-4bf6-8152-5fd4e4957e91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:43:40.574168 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:40.574011 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7c1bd6c1-4939-4bf6-8152-5fd4e4957e91" (UID: "7c1bd6c1-4939-4bf6-8152-5fd4e4957e91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:43:40.574168 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:40.574028 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7c1bd6c1-4939-4bf6-8152-5fd4e4957e91" (UID: "7c1bd6c1-4939-4bf6-8152-5fd4e4957e91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:43:40.574480 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:40.574358 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7c1bd6c1-4939-4bf6-8152-5fd4e4957e91" (UID: "7c1bd6c1-4939-4bf6-8152-5fd4e4957e91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:43:40.574916 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:40.574888 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-istio-data" (OuterVolumeSpecName: "istio-data") pod "7c1bd6c1-4939-4bf6-8152-5fd4e4957e91" (UID: "7c1bd6c1-4939-4bf6-8152-5fd4e4957e91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:43:40.576073 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:40.576046 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7c1bd6c1-4939-4bf6-8152-5fd4e4957e91" (UID: "7c1bd6c1-4939-4bf6-8152-5fd4e4957e91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:43:40.576289 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:40.576266 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7c1bd6c1-4939-4bf6-8152-5fd4e4957e91" (UID: "7c1bd6c1-4939-4bf6-8152-5fd4e4957e91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 18:43:40.576391 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:40.576370 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-kube-api-access-vc79c" (OuterVolumeSpecName: "kube-api-access-vc79c") pod "7c1bd6c1-4939-4bf6-8152-5fd4e4957e91" (UID: "7c1bd6c1-4939-4bf6-8152-5fd4e4957e91"). InnerVolumeSpecName "kube-api-access-vc79c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:43:40.576697 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:40.576678 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-istio-token" (OuterVolumeSpecName: "istio-token") pod "7c1bd6c1-4939-4bf6-8152-5fd4e4957e91" (UID: "7c1bd6c1-4939-4bf6-8152-5fd4e4957e91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:43:40.674523 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:40.674486 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-credential-socket\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:43:40.674523 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:40.674520 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-istio-data\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:43:40.674523 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:40.674531 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-istiod-ca-cert\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:43:40.674523 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:40.674539 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-istio-podinfo\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:43:40.674832 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:40.674546 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-istio-token\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:43:40.674832 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:40.674554 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc79c\" (UniqueName: \"kubernetes.io/projected/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-kube-api-access-vc79c\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:43:40.674832 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:40.674562 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-istio-envoy\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:43:40.674832 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:40.674571 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-workload-socket\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:43:40.674832 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:40.674578 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91-workload-certs\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:43:41.026021 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:41.025989 2565 generic.go:356] "Generic (PLEG): container finished" podID="7c1bd6c1-4939-4bf6-8152-5fd4e4957e91" containerID="4c0a1d7f2c857d5919800cf66c40efc3aebb393ffb3d1399b1fb8f4fa33ec621" exitCode=0 Jul 14 18:43:41.026325 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:41.026026 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" event={"ID":"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91","Type":"ContainerDied","Data":"4c0a1d7f2c857d5919800cf66c40efc3aebb393ffb3d1399b1fb8f4fa33ec621"} Jul 14 18:43:41.026325 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:41.026033 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" Jul 14 18:43:41.026325 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:41.026069 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5" event={"ID":"7c1bd6c1-4939-4bf6-8152-5fd4e4957e91","Type":"ContainerDied","Data":"ba2037bd16b0d370c711abbf520201be7fe13f61c5de042b8c076264da9ca0ba"} Jul 14 18:43:41.026325 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:41.026092 2565 scope.go:117] "RemoveContainer" containerID="4c0a1d7f2c857d5919800cf66c40efc3aebb393ffb3d1399b1fb8f4fa33ec621" Jul 14 18:43:41.035579 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:41.035560 2565 scope.go:117] "RemoveContainer" containerID="fc20195997d671f86460bdb3676a5ccb972329a51406ba2697462325aa86fdf2" Jul 14 18:43:41.043421 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:41.043406 2565 scope.go:117] "RemoveContainer" containerID="4c0a1d7f2c857d5919800cf66c40efc3aebb393ffb3d1399b1fb8f4fa33ec621" Jul 14 18:43:41.043643 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:43:41.043623 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c0a1d7f2c857d5919800cf66c40efc3aebb393ffb3d1399b1fb8f4fa33ec621\": container with ID starting with 4c0a1d7f2c857d5919800cf66c40efc3aebb393ffb3d1399b1fb8f4fa33ec621 not found: ID does not exist" containerID="4c0a1d7f2c857d5919800cf66c40efc3aebb393ffb3d1399b1fb8f4fa33ec621" Jul 14 18:43:41.043716 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:41.043655 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c0a1d7f2c857d5919800cf66c40efc3aebb393ffb3d1399b1fb8f4fa33ec621"} err="failed to get container status \"4c0a1d7f2c857d5919800cf66c40efc3aebb393ffb3d1399b1fb8f4fa33ec621\": rpc error: code = NotFound desc = could not find container \"4c0a1d7f2c857d5919800cf66c40efc3aebb393ffb3d1399b1fb8f4fa33ec621\": container with ID starting with 4c0a1d7f2c857d5919800cf66c40efc3aebb393ffb3d1399b1fb8f4fa33ec621 not found: ID does not exist" Jul 14 18:43:41.043716 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:41.043681 2565 scope.go:117] "RemoveContainer" containerID="fc20195997d671f86460bdb3676a5ccb972329a51406ba2697462325aa86fdf2" Jul 14 18:43:41.043916 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:43:41.043900 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc20195997d671f86460bdb3676a5ccb972329a51406ba2697462325aa86fdf2\": container with ID starting with fc20195997d671f86460bdb3676a5ccb972329a51406ba2697462325aa86fdf2 not found: ID does not exist" containerID="fc20195997d671f86460bdb3676a5ccb972329a51406ba2697462325aa86fdf2" Jul 14 18:43:41.043978 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:41.043923 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc20195997d671f86460bdb3676a5ccb972329a51406ba2697462325aa86fdf2"} err="failed to get container status \"fc20195997d671f86460bdb3676a5ccb972329a51406ba2697462325aa86fdf2\": rpc error: code = NotFound desc = could not find container \"fc20195997d671f86460bdb3676a5ccb972329a51406ba2697462325aa86fdf2\": container with ID starting with fc20195997d671f86460bdb3676a5ccb972329a51406ba2697462325aa86fdf2 not found: ID does not exist" Jul 14 18:43:41.050972 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:41.050950 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5"] Jul 14 18:43:41.058057 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:41.058037 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c99ca1c1-openshift-default-c8cfdf57f-d5wd5"] Jul 14 18:43:41.163404 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:41.163374 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:43:41.693175 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:41.692949 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c1bd6c1-4939-4bf6-8152-5fd4e4957e91" path="/var/lib/kubelet/pods/7c1bd6c1-4939-4bf6-8152-5fd4e4957e91/volumes" Jul 14 18:43:54.207233 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:54.207193 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:43:55.831005 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:43:55.830971 2565 scope.go:117] "RemoveContainer" containerID="98b8266fb7af91ef4a5e0f308f511a2dc7519097e7d5db7ddfebe6b3ae445795" Jul 14 18:44:35.971622 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:35.971524 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5"] Jul 14 18:44:35.972240 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:35.972048 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c1bd6c1-4939-4bf6-8152-5fd4e4957e91" containerName="istio-proxy" Jul 14 18:44:35.972240 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:35.972067 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c1bd6c1-4939-4bf6-8152-5fd4e4957e91" containerName="istio-proxy" Jul 14 18:44:35.972240 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:35.972125 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c1bd6c1-4939-4bf6-8152-5fd4e4957e91" containerName="istio-proxy" Jul 14 18:44:35.972240 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:35.972134 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c1bd6c1-4939-4bf6-8152-5fd4e4957e91" containerName="istio-proxy" Jul 14 18:44:35.972240 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:35.972148 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21a9f60c-0c98-4374-a7e1-95f35a4a6c66" containerName="revoke-keys" Jul 14 18:44:35.972240 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:35.972157 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="21a9f60c-0c98-4374-a7e1-95f35a4a6c66" containerName="revoke-keys" Jul 14 18:44:35.972240 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:35.972242 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c1bd6c1-4939-4bf6-8152-5fd4e4957e91" containerName="istio-proxy" Jul 14 18:44:35.972617 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:35.972256 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="21a9f60c-0c98-4374-a7e1-95f35a4a6c66" containerName="revoke-keys" Jul 14 18:44:35.975554 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:35.975532 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:35.978075 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:35.978054 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1f114d-openshift-default-dockercfg-v4p9s\"" Jul 14 18:44:35.989904 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:35.989878 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5"] Jul 14 18:44:36.036892 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.036858 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c90c4714-014c-47a4-9377-47fc83be7dc5-workload-certs\") pod \"e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:36.037087 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.036931 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c90c4714-014c-47a4-9377-47fc83be7dc5-istiod-ca-cert\") pod \"e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:36.037087 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.036977 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c90c4714-014c-47a4-9377-47fc83be7dc5-istio-token\") pod \"e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:36.037087 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.037031 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4x8n\" (UniqueName: \"kubernetes.io/projected/c90c4714-014c-47a4-9377-47fc83be7dc5-kube-api-access-f4x8n\") pod \"e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:36.037087 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.037055 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c90c4714-014c-47a4-9377-47fc83be7dc5-workload-socket\") pod \"e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:36.037259 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.037143 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c90c4714-014c-47a4-9377-47fc83be7dc5-istio-envoy\") pod \"e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:36.037259 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.037198 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c90c4714-014c-47a4-9377-47fc83be7dc5-credential-socket\") pod \"e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:36.037259 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.037219 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c90c4714-014c-47a4-9377-47fc83be7dc5-istio-podinfo\") pod \"e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:36.037259 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.037245 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c90c4714-014c-47a4-9377-47fc83be7dc5-istio-data\") pod \"e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:36.137938 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.137901 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4x8n\" (UniqueName: \"kubernetes.io/projected/c90c4714-014c-47a4-9377-47fc83be7dc5-kube-api-access-f4x8n\") pod \"e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:36.137938 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.137940 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c90c4714-014c-47a4-9377-47fc83be7dc5-workload-socket\") pod \"e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:36.138226 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.137966 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c90c4714-014c-47a4-9377-47fc83be7dc5-istio-envoy\") pod \"e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:36.138226 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.137997 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c90c4714-014c-47a4-9377-47fc83be7dc5-credential-socket\") pod \"e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:36.138226 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.138013 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c90c4714-014c-47a4-9377-47fc83be7dc5-istio-podinfo\") pod \"e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:36.138226 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.138037 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c90c4714-014c-47a4-9377-47fc83be7dc5-istio-data\") pod \"e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:36.138226 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.138053 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c90c4714-014c-47a4-9377-47fc83be7dc5-workload-certs\") pod \"e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:36.138226 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.138126 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c90c4714-014c-47a4-9377-47fc83be7dc5-istiod-ca-cert\") pod \"e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:36.138226 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.138156 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c90c4714-014c-47a4-9377-47fc83be7dc5-istio-token\") pod \"e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:36.138572 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.138396 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c90c4714-014c-47a4-9377-47fc83be7dc5-workload-socket\") pod \"e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:36.138572 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.138480 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c90c4714-014c-47a4-9377-47fc83be7dc5-istio-data\") pod \"e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:36.138572 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.138499 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c90c4714-014c-47a4-9377-47fc83be7dc5-credential-socket\") pod \"e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:36.138572 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.138546 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c90c4714-014c-47a4-9377-47fc83be7dc5-workload-certs\") pod \"e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:36.138951 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.138933 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c90c4714-014c-47a4-9377-47fc83be7dc5-istiod-ca-cert\") pod \"e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:36.140518 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.140499 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c90c4714-014c-47a4-9377-47fc83be7dc5-istio-envoy\") pod \"e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:36.140859 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.140838 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c90c4714-014c-47a4-9377-47fc83be7dc5-istio-podinfo\") pod \"e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:36.148966 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.148940 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c90c4714-014c-47a4-9377-47fc83be7dc5-istio-token\") pod \"e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:36.149154 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.149139 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4x8n\" (UniqueName: \"kubernetes.io/projected/c90c4714-014c-47a4-9377-47fc83be7dc5-kube-api-access-f4x8n\") pod \"e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:36.285973 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.285945 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:36.439679 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.439653 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5"] Jul 14 18:44:36.442194 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:44:36.442169 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc90c4714_014c_47a4_9377_47fc83be7dc5.slice/crio-a431824c932034ab4a33e29c2b20402518e2d631cbab1efc04a756f3021bc2c7 WatchSource:0}: Error finding container a431824c932034ab4a33e29c2b20402518e2d631cbab1efc04a756f3021bc2c7: Status 404 returned error can't find the container with id a431824c932034ab4a33e29c2b20402518e2d631cbab1efc04a756f3021bc2c7 Jul 14 18:44:36.444158 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.444120 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:44:36.444269 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.444186 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:44:36.444269 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:36.444217 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:44:37.290920 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:37.290870 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" event={"ID":"c90c4714-014c-47a4-9377-47fc83be7dc5","Type":"ContainerStarted","Data":"5d6b68de9262bc318fcff9d7f91db7e7d89bdc94839d35eaa63d35d840e45ffc"} Jul 14 18:44:37.291443 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:37.290978 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" event={"ID":"c90c4714-014c-47a4-9377-47fc83be7dc5","Type":"ContainerStarted","Data":"a431824c932034ab4a33e29c2b20402518e2d631cbab1efc04a756f3021bc2c7"} Jul 14 18:44:37.317462 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:37.317399 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" podStartSLOduration=2.317383377 podStartE2EDuration="2.317383377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:44:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:44:37.315738415 +0000 UTC m=+2082.153081697" watchObservedRunningTime="2026-07-14 18:44:37.317383377 +0000 UTC m=+2082.154726633" Jul 14 18:44:38.287045 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:38.286984 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:38.293404 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:38.293373 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:38.296625 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:38.296603 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:38.297566 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:38.297520 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:47.351961 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:47.351920 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:44:54.338316 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:54.338273 2565 patch_prober.go:28] interesting pod/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 14 18:44:54.338737 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:54.338344 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" podUID="c90c4714-014c-47a4-9377-47fc83be7dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 14 18:44:55.397687 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:55.397578 2565 generic.go:356] "Generic (PLEG): container finished" podID="c90c4714-014c-47a4-9377-47fc83be7dc5" containerID="5d6b68de9262bc318fcff9d7f91db7e7d89bdc94839d35eaa63d35d840e45ffc" exitCode=0 Jul 14 18:44:55.397687 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:55.397624 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" event={"ID":"c90c4714-014c-47a4-9377-47fc83be7dc5","Type":"ContainerDied","Data":"5d6b68de9262bc318fcff9d7f91db7e7d89bdc94839d35eaa63d35d840e45ffc"} Jul 14 18:44:55.398235 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:55.398158 2565 scope.go:117] "RemoveContainer" containerID="5d6b68de9262bc318fcff9d7f91db7e7d89bdc94839d35eaa63d35d840e45ffc" Jul 14 18:44:55.399279 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:55.399249 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:44:55.399366 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:55.399321 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:44:55.399366 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:55.399354 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:44:55.768248 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:55.768219 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wqpt2_4525dcda-ead6-4800-8d17-58812c289010/console-operator/1.log" Jul 14 18:44:55.769238 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:55.769062 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wqpt2_4525dcda-ead6-4800-8d17-58812c289010/console-operator/1.log" Jul 14 18:44:55.779155 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:55.779129 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f99bz_5f0ecb09-5c7f-4709-9b4a-1a6a03580822/ovn-acl-logging/0.log" Jul 14 18:44:55.779301 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:55.779277 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f99bz_5f0ecb09-5c7f-4709-9b4a-1a6a03580822/ovn-acl-logging/0.log" Jul 14 18:44:56.409300 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:56.409253 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" event={"ID":"c90c4714-014c-47a4-9377-47fc83be7dc5","Type":"ContainerStarted","Data":"d4260e15e63e2128b42eb4179f1dce7fe0d862907163919c127398c00602a07a"} Jul 14 18:44:57.287222 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:57.287120 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:44:57.287454 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:57.287421 2565 patch_prober.go:28] interesting pod/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 14 18:44:57.287676 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:57.287646 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" podUID="c90c4714-014c-47a4-9377-47fc83be7dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 14 18:44:58.287396 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:58.287357 2565 patch_prober.go:28] interesting pod/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 14 18:44:58.287921 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:58.287429 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" podUID="c90c4714-014c-47a4-9377-47fc83be7dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 14 18:44:59.286984 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:59.286933 2565 patch_prober.go:28] interesting pod/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 14 18:44:59.287229 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:44:59.287006 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" podUID="c90c4714-014c-47a4-9377-47fc83be7dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 14 18:45:00.137626 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:00.137582 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7965f3-29734245-4n7b8"] Jul 14 18:45:00.138274 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:00.138255 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c1bd6c1-4939-4bf6-8152-5fd4e4957e91" containerName="istio-proxy" Jul 14 18:45:00.141514 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:00.141496 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f114d-29734245-wvqw2"] Jul 14 18:45:00.141638 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:00.141618 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7965f3-29734245-4n7b8" Jul 14 18:45:00.144511 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:00.144488 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734245-26cjg"] Jul 14 18:45:00.144652 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:00.144519 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9jplk\"" Jul 14 18:45:00.144652 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:00.144614 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f114d-29734245-wvqw2" Jul 14 18:45:00.147372 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:00.147352 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734245-26cjg" Jul 14 18:45:00.152612 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:00.152586 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734245-26cjg"] Jul 14 18:45:00.154136 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:00.154087 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7965f3-29734245-4n7b8" Jul 14 18:45:00.155546 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:00.155527 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f114d-29734245-wvqw2"] Jul 14 18:45:00.166504 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:00.166473 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7965f3-29734245-4n7b8"] Jul 14 18:45:00.169979 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:00.169955 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f114d-29734245-wvqw2" Jul 14 18:45:00.236143 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:00.236110 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734245-26cjg" Jul 14 18:45:00.288637 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:00.288597 2565 patch_prober.go:28] interesting pod/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 14 18:45:00.288810 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:00.288663 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" podUID="c90c4714-014c-47a4-9377-47fc83be7dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 14 18:45:00.423452 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:00.423424 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734245-26cjg"] Jul 14 18:45:00.425462 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:45:00.425435 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03021d3_89fb_4bc0_91cd_b7fa20c16e3c.slice/crio-3603d38c22130a08ed6033b55479e773b333fcf241fc4e5f6d4a476a3da6f8dc WatchSource:0}: Error finding container 3603d38c22130a08ed6033b55479e773b333fcf241fc4e5f6d4a476a3da6f8dc: Status 404 returned error can't find the container with id 3603d38c22130a08ed6033b55479e773b333fcf241fc4e5f6d4a476a3da6f8dc Jul 14 18:45:00.583932 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:00.583899 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7965f3-29734245-4n7b8"] Jul 14 18:45:00.589711 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:00.589668 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f114d-29734245-wvqw2"] Jul 14 18:45:00.592121 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:45:00.591813 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod720bf2b9_8764_49db_83bb_09d42d2d5e1e.slice/crio-0b9fcaeb88108084ba51b1634e6dc96359b55eda0382a95390d1d54be64ebb30 WatchSource:0}: Error finding container 0b9fcaeb88108084ba51b1634e6dc96359b55eda0382a95390d1d54be64ebb30: Status 404 returned error can't find the container with id 0b9fcaeb88108084ba51b1634e6dc96359b55eda0382a95390d1d54be64ebb30 Jul 14 18:45:01.291885 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:01.291842 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:45:01.292340 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:01.292274 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:45:01.293047 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:01.293027 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:45:01.436153 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:01.436030 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734245-26cjg" event={"ID":"f03021d3-89fb-4bc0-91cd-b7fa20c16e3c","Type":"ContainerStarted","Data":"2082c21a2a8ca92bd62ed3dd2e4da771821e18cc8c857bd23d3fa19b6032fb96"} Jul 14 18:45:01.436153 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:01.436079 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734245-26cjg" event={"ID":"f03021d3-89fb-4bc0-91cd-b7fa20c16e3c","Type":"ContainerStarted","Data":"3603d38c22130a08ed6033b55479e773b333fcf241fc4e5f6d4a476a3da6f8dc"} Jul 14 18:45:01.437574 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:01.437548 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f114d-29734245-wvqw2" event={"ID":"720bf2b9-8764-49db-83bb-09d42d2d5e1e","Type":"ContainerStarted","Data":"dd4a058f2f33d9f7c2db5d6d2dbefe040ac366a666bfab68989ae4a3930e7db2"} Jul 14 18:45:01.437691 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:01.437591 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f114d-29734245-wvqw2" event={"ID":"720bf2b9-8764-49db-83bb-09d42d2d5e1e","Type":"ContainerStarted","Data":"0b9fcaeb88108084ba51b1634e6dc96359b55eda0382a95390d1d54be64ebb30"} Jul 14 18:45:01.439025 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:01.439006 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7965f3-29734245-4n7b8" event={"ID":"ec5fce56-a1be-44eb-a58c-91f1d7eaff5a","Type":"ContainerStarted","Data":"13e196df360a8463f0c0e5b773ba2a45e152d7eebc2db5f1403fedadc309760d"} Jul 14 18:45:01.439162 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:01.439036 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7965f3-29734245-4n7b8" event={"ID":"ec5fce56-a1be-44eb-a58c-91f1d7eaff5a","Type":"ContainerStarted","Data":"fdfac0b93d1c73de98768de5bbfabf660ff67cfa7564d69c10fff58cdf2bfaa1"} Jul 14 18:45:01.455523 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:01.455470 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734245-26cjg" podStartSLOduration=1.4554545509999999 podStartE2EDuration="1.455454551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:45:01.453080929 +0000 UTC m=+2106.290424186" watchObservedRunningTime="2026-07-14 18:45:01.455454551 +0000 UTC m=+2106.292797809" Jul 14 18:45:01.470320 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:01.470267 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f114d-29734245-wvqw2" podStartSLOduration=1.4702533230000001 podStartE2EDuration="1.470253323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:45:01.469883206 +0000 UTC m=+2106.307226465" watchObservedRunningTime="2026-07-14 18:45:01.470253323 +0000 UTC m=+2106.307596582" Jul 14 18:45:01.489116 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:01.489040 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7965f3-29734245-4n7b8" podStartSLOduration=1.489021054 podStartE2EDuration="1.489021054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:45:01.486928335 +0000 UTC m=+2106.324271596" watchObservedRunningTime="2026-07-14 18:45:01.489021054 +0000 UTC m=+2106.326364312" Jul 14 18:45:02.250871 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:02.250775 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:45:02.445220 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:02.445185 2565 generic.go:356] "Generic (PLEG): container finished" podID="720bf2b9-8764-49db-83bb-09d42d2d5e1e" containerID="dd4a058f2f33d9f7c2db5d6d2dbefe040ac366a666bfab68989ae4a3930e7db2" exitCode=0 Jul 14 18:45:02.445665 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:02.445274 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f114d-29734245-wvqw2" event={"ID":"720bf2b9-8764-49db-83bb-09d42d2d5e1e","Type":"ContainerDied","Data":"dd4a058f2f33d9f7c2db5d6d2dbefe040ac366a666bfab68989ae4a3930e7db2"} Jul 14 18:45:02.446856 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:02.446826 2565 generic.go:356] "Generic (PLEG): container finished" podID="ec5fce56-a1be-44eb-a58c-91f1d7eaff5a" containerID="13e196df360a8463f0c0e5b773ba2a45e152d7eebc2db5f1403fedadc309760d" exitCode=0 Jul 14 18:45:02.446962 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:02.446914 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7965f3-29734245-4n7b8" event={"ID":"ec5fce56-a1be-44eb-a58c-91f1d7eaff5a","Type":"ContainerDied","Data":"13e196df360a8463f0c0e5b773ba2a45e152d7eebc2db5f1403fedadc309760d"} Jul 14 18:45:02.448411 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:02.448386 2565 generic.go:356] "Generic (PLEG): container finished" podID="f03021d3-89fb-4bc0-91cd-b7fa20c16e3c" containerID="2082c21a2a8ca92bd62ed3dd2e4da771821e18cc8c857bd23d3fa19b6032fb96" exitCode=0 Jul 14 18:45:02.448526 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:02.448445 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734245-26cjg" event={"ID":"f03021d3-89fb-4bc0-91cd-b7fa20c16e3c","Type":"ContainerDied","Data":"2082c21a2a8ca92bd62ed3dd2e4da771821e18cc8c857bd23d3fa19b6032fb96"} Jul 14 18:45:03.611833 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:03.611809 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7965f3-29734245-4n7b8" Jul 14 18:45:03.656159 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:03.656131 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734245-26cjg" Jul 14 18:45:03.659161 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:03.659140 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f114d-29734245-wvqw2" Jul 14 18:45:04.460088 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:04.460051 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734245-26cjg" Jul 14 18:45:04.460088 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:04.460053 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734245-26cjg" event={"ID":"f03021d3-89fb-4bc0-91cd-b7fa20c16e3c","Type":"ContainerDied","Data":"3603d38c22130a08ed6033b55479e773b333fcf241fc4e5f6d4a476a3da6f8dc"} Jul 14 18:45:04.460379 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:04.460196 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3603d38c22130a08ed6033b55479e773b333fcf241fc4e5f6d4a476a3da6f8dc" Jul 14 18:45:04.461448 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:04.461428 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f114d-29734245-wvqw2" Jul 14 18:45:04.461557 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:04.461445 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f114d-29734245-wvqw2" event={"ID":"720bf2b9-8764-49db-83bb-09d42d2d5e1e","Type":"ContainerDied","Data":"0b9fcaeb88108084ba51b1634e6dc96359b55eda0382a95390d1d54be64ebb30"} Jul 14 18:45:04.461557 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:04.461493 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b9fcaeb88108084ba51b1634e6dc96359b55eda0382a95390d1d54be64ebb30" Jul 14 18:45:04.463060 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:04.463046 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7965f3-29734245-4n7b8" Jul 14 18:45:04.463185 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:04.463057 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7965f3-29734245-4n7b8" event={"ID":"ec5fce56-a1be-44eb-a58c-91f1d7eaff5a","Type":"ContainerDied","Data":"fdfac0b93d1c73de98768de5bbfabf660ff67cfa7564d69c10fff58cdf2bfaa1"} Jul 14 18:45:04.463185 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:04.463121 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdfac0b93d1c73de98768de5bbfabf660ff67cfa7564d69c10fff58cdf2bfaa1" Jul 14 18:45:05.074785 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.074746 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x"] Jul 14 18:45:05.075211 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.075090 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec5fce56-a1be-44eb-a58c-91f1d7eaff5a" containerName="cleanup" Jul 14 18:45:05.075211 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.075118 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec5fce56-a1be-44eb-a58c-91f1d7eaff5a" containerName="cleanup" Jul 14 18:45:05.075211 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.075137 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f03021d3-89fb-4bc0-91cd-b7fa20c16e3c" containerName="cleanup" Jul 14 18:45:05.075211 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.075147 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f03021d3-89fb-4bc0-91cd-b7fa20c16e3c" containerName="cleanup" Jul 14 18:45:05.075211 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.075159 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="720bf2b9-8764-49db-83bb-09d42d2d5e1e" containerName="cleanup" Jul 14 18:45:05.075211 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.075165 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="720bf2b9-8764-49db-83bb-09d42d2d5e1e" containerName="cleanup" Jul 14 18:45:05.075431 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.075221 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f03021d3-89fb-4bc0-91cd-b7fa20c16e3c" containerName="cleanup" Jul 14 18:45:05.075431 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.075232 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec5fce56-a1be-44eb-a58c-91f1d7eaff5a" containerName="cleanup" Jul 14 18:45:05.075431 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.075238 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="720bf2b9-8764-49db-83bb-09d42d2d5e1e" containerName="cleanup" Jul 14 18:45:05.078153 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.078136 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" Jul 14 18:45:05.081951 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.081920 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1f114d\"/\"default-dockercfg-lbtm2\"" Jul 14 18:45:05.081951 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.081947 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1f114d\"/\"test-model-1f114d-kserve-self-signed-certs\"" Jul 14 18:45:05.082178 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.081971 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1f114d\"/\"openshift-service-ca.crt\"" Jul 14 18:45:05.082178 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.082007 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1f114d\"/\"kube-root-ca.crt\"" Jul 14 18:45:05.090244 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.090218 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x"] Jul 14 18:45:05.112287 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.112254 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-tmp-dir\") pod \"test-model-1f114d-kserve-855c78fd7f-2q55x\" (UID: \"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d\") " pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" Jul 14 18:45:05.112287 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.112295 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-tls-certs\") pod \"test-model-1f114d-kserve-855c78fd7f-2q55x\" (UID: \"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d\") " pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" Jul 14 18:45:05.112498 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.112375 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-dshm\") pod \"test-model-1f114d-kserve-855c78fd7f-2q55x\" (UID: \"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d\") " pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" Jul 14 18:45:05.112498 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.112421 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rsdg\" (UniqueName: \"kubernetes.io/projected/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-kube-api-access-5rsdg\") pod \"test-model-1f114d-kserve-855c78fd7f-2q55x\" (UID: \"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d\") " pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" Jul 14 18:45:05.112613 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.112502 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-home\") pod \"test-model-1f114d-kserve-855c78fd7f-2q55x\" (UID: \"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d\") " pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" Jul 14 18:45:05.112613 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.112529 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-model-cache\") pod \"test-model-1f114d-kserve-855c78fd7f-2q55x\" (UID: \"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d\") " pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" Jul 14 18:45:05.212953 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.212913 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-model-cache\") pod \"test-model-1f114d-kserve-855c78fd7f-2q55x\" (UID: \"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d\") " pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" Jul 14 18:45:05.213183 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.212979 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-tmp-dir\") pod \"test-model-1f114d-kserve-855c78fd7f-2q55x\" (UID: \"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d\") " pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" Jul 14 18:45:05.213183 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.213002 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-tls-certs\") pod \"test-model-1f114d-kserve-855c78fd7f-2q55x\" (UID: \"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d\") " pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" Jul 14 18:45:05.213183 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.213046 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-dshm\") pod \"test-model-1f114d-kserve-855c78fd7f-2q55x\" (UID: \"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d\") " pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" Jul 14 18:45:05.213183 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.213079 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rsdg\" (UniqueName: \"kubernetes.io/projected/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-kube-api-access-5rsdg\") pod \"test-model-1f114d-kserve-855c78fd7f-2q55x\" (UID: \"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d\") " pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" Jul 14 18:45:05.213183 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.213177 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-home\") pod \"test-model-1f114d-kserve-855c78fd7f-2q55x\" (UID: \"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d\") " pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" Jul 14 18:45:05.213432 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.213400 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-model-cache\") pod \"test-model-1f114d-kserve-855c78fd7f-2q55x\" (UID: \"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d\") " pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" Jul 14 18:45:05.213483 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.213452 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-tmp-dir\") pod \"test-model-1f114d-kserve-855c78fd7f-2q55x\" (UID: \"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d\") " pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" Jul 14 18:45:05.213533 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.213502 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-home\") pod \"test-model-1f114d-kserve-855c78fd7f-2q55x\" (UID: \"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d\") " pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" Jul 14 18:45:05.215560 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.215539 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-dshm\") pod \"test-model-1f114d-kserve-855c78fd7f-2q55x\" (UID: \"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d\") " pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" Jul 14 18:45:05.215852 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.215820 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-tls-certs\") pod \"test-model-1f114d-kserve-855c78fd7f-2q55x\" (UID: \"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d\") " pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" Jul 14 18:45:05.224790 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.224758 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rsdg\" (UniqueName: \"kubernetes.io/projected/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-kube-api-access-5rsdg\") pod \"test-model-1f114d-kserve-855c78fd7f-2q55x\" (UID: \"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d\") " pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" Jul 14 18:45:05.388604 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.388504 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" Jul 14 18:45:05.541335 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:05.541308 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x"] Jul 14 18:45:05.542934 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:45:05.542908 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f3d13dc_b43a_484d_87a2_5f2eab6a6a9d.slice/crio-71c3fa008f8ad943a383436c9be00e1914d8240cdb80c20d52663a8a2e949fb4 WatchSource:0}: Error finding container 71c3fa008f8ad943a383436c9be00e1914d8240cdb80c20d52663a8a2e949fb4: Status 404 returned error can't find the container with id 71c3fa008f8ad943a383436c9be00e1914d8240cdb80c20d52663a8a2e949fb4 Jul 14 18:45:06.474798 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:06.474759 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" event={"ID":"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d","Type":"ContainerStarted","Data":"5382d7e1fa0fb30ac7fcaa3b0c04e7f8c7e9d03589826d477060f06c8c1b7e93"} Jul 14 18:45:06.475277 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:06.474810 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" event={"ID":"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d","Type":"ContainerStarted","Data":"71c3fa008f8ad943a383436c9be00e1914d8240cdb80c20d52663a8a2e949fb4"} Jul 14 18:45:06.494946 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:06.494892 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" podStartSLOduration=1.494875325 podStartE2EDuration="1.494875325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:45:06.492887014 +0000 UTC m=+2111.330230270" watchObservedRunningTime="2026-07-14 18:45:06.494875325 +0000 UTC m=+2111.332218583" Jul 14 18:45:15.067493 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:15.067454 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:45:15.389500 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:15.389395 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" Jul 14 18:45:15.389671 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:15.389553 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" Jul 14 18:45:15.402453 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:15.402427 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" Jul 14 18:45:15.534733 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:15.534698 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" Jul 14 18:45:43.257041 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:45:43.257008 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:46:12.052679 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:12.052644 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x"] Jul 14 18:46:13.183328 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:13.183287 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f114d-2qsqw"] Jul 14 18:46:13.186710 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:13.186673 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f114d-2qsqw" Jul 14 18:46:13.189646 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:13.189618 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 18:46:13.189801 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:13.189774 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9jplk\"" Jul 14 18:46:13.194395 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:13.194361 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f114d-2qsqw"] Jul 14 18:46:13.309493 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:13.309456 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c6ca669-5fc0-403c-8627-10ad92528262-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f114d-2qsqw\" (UID: \"4c6ca669-5fc0-403c-8627-10ad92528262\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f114d-2qsqw" Jul 14 18:46:13.410380 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:13.410340 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c6ca669-5fc0-403c-8627-10ad92528262-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f114d-2qsqw\" (UID: \"4c6ca669-5fc0-403c-8627-10ad92528262\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f114d-2qsqw" Jul 14 18:46:13.411001 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:13.410981 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c6ca669-5fc0-403c-8627-10ad92528262-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f114d-2qsqw\" (UID: \"4c6ca669-5fc0-403c-8627-10ad92528262\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f114d-2qsqw" Jul 14 18:46:13.500770 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:13.500683 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f114d-2qsqw" Jul 14 18:46:13.662682 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:13.662656 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f114d-2qsqw"] Jul 14 18:46:13.662850 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:46:13.662757 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c6ca669_5fc0_403c_8627_10ad92528262.slice/crio-09181f048467662c56e42096be7f032a40287574355e0deb9965ae7424e0af84 WatchSource:0}: Error finding container 09181f048467662c56e42096be7f032a40287574355e0deb9965ae7424e0af84: Status 404 returned error can't find the container with id 09181f048467662c56e42096be7f032a40287574355e0deb9965ae7424e0af84 Jul 14 18:46:13.818287 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:13.818250 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f114d-2qsqw" event={"ID":"4c6ca669-5fc0-403c-8627-10ad92528262","Type":"ContainerStarted","Data":"09181f048467662c56e42096be7f032a40287574355e0deb9965ae7424e0af84"} Jul 14 18:46:14.825675 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:14.825636 2565 generic.go:356] "Generic (PLEG): container finished" podID="4c6ca669-5fc0-403c-8627-10ad92528262" containerID="f7df03dc99344f28bcd79af0ce9d9721d747467c8764622c27484d04466da471" exitCode=0 Jul 14 18:46:14.826070 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:14.825702 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f114d-2qsqw" event={"ID":"4c6ca669-5fc0-403c-8627-10ad92528262","Type":"ContainerDied","Data":"f7df03dc99344f28bcd79af0ce9d9721d747467c8764622c27484d04466da471"} Jul 14 18:46:15.970661 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:15.970633 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f114d-2qsqw" Jul 14 18:46:16.033539 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:16.033499 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c6ca669-5fc0-403c-8627-10ad92528262-maas-api-service-ca\") pod \"4c6ca669-5fc0-403c-8627-10ad92528262\" (UID: \"4c6ca669-5fc0-403c-8627-10ad92528262\") " Jul 14 18:46:16.033874 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:16.033852 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c6ca669-5fc0-403c-8627-10ad92528262-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4c6ca669-5fc0-403c-8627-10ad92528262" (UID: "4c6ca669-5fc0-403c-8627-10ad92528262"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:46:16.135209 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:16.135093 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c6ca669-5fc0-403c-8627-10ad92528262-maas-api-service-ca\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:46:16.836466 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:16.836433 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f114d-2qsqw" Jul 14 18:46:16.836466 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:16.836459 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f114d-2qsqw" event={"ID":"4c6ca669-5fc0-403c-8627-10ad92528262","Type":"ContainerDied","Data":"09181f048467662c56e42096be7f032a40287574355e0deb9965ae7424e0af84"} Jul 14 18:46:16.836734 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:16.836508 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09181f048467662c56e42096be7f032a40287574355e0deb9965ae7424e0af84" Jul 14 18:46:17.353686 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:17.353647 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:46:23.790594 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:23.790552 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f114d-29734245-wvqw2"] Jul 14 18:46:23.793712 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:23.793681 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f114d-29734245-wvqw2"] Jul 14 18:46:25.698002 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:25.697962 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="720bf2b9-8764-49db-83bb-09d42d2d5e1e" path="/var/lib/kubelet/pods/720bf2b9-8764-49db-83bb-09d42d2d5e1e/volumes" Jul 14 18:46:27.073552 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:27.073504 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" podUID="6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d" containerName="main" containerID="cri-o://5382d7e1fa0fb30ac7fcaa3b0c04e7f8c7e9d03589826d477060f06c8c1b7e93" gracePeriod=45 Jul 14 18:46:27.332131 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:27.332056 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" Jul 14 18:46:27.440718 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:27.440684 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-dshm\") pod \"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d\" (UID: \"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d\") " Jul 14 18:46:27.440894 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:27.440737 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-tmp-dir\") pod \"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d\" (UID: \"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d\") " Jul 14 18:46:27.440894 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:27.440767 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-home\") pod \"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d\" (UID: \"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d\") " Jul 14 18:46:27.440894 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:27.440860 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-tls-certs\") pod \"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d\" (UID: \"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d\") " Jul 14 18:46:27.440894 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:27.440888 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-model-cache\") pod \"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d\" (UID: \"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d\") " Jul 14 18:46:27.441057 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:27.440937 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rsdg\" (UniqueName: \"kubernetes.io/projected/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-kube-api-access-5rsdg\") pod \"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d\" (UID: \"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d\") " Jul 14 18:46:27.441145 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:27.441063 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d" (UID: "6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:46:27.441145 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:27.441081 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-home" (OuterVolumeSpecName: "home") pod "6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d" (UID: "6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:46:27.441257 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:27.441197 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-model-cache" (OuterVolumeSpecName: "model-cache") pod "6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d" (UID: "6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:46:27.441378 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:27.441300 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-model-cache\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:46:27.441378 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:27.441336 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-tmp-dir\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:46:27.441378 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:27.441344 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-home\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:46:27.443261 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:27.443234 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d" (UID: "6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:46:27.443364 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:27.443278 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-kube-api-access-5rsdg" (OuterVolumeSpecName: "kube-api-access-5rsdg") pod "6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d" (UID: "6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d"). InnerVolumeSpecName "kube-api-access-5rsdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:46:27.443364 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:27.443296 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-dshm" (OuterVolumeSpecName: "dshm") pod "6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d" (UID: "6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:46:27.542241 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:27.542203 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rsdg\" (UniqueName: \"kubernetes.io/projected/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-kube-api-access-5rsdg\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:46:27.542241 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:27.542234 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-dshm\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:46:27.542241 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:27.542246 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d-tls-certs\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:46:27.896801 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:27.896761 2565 generic.go:356] "Generic (PLEG): container finished" podID="6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d" containerID="5382d7e1fa0fb30ac7fcaa3b0c04e7f8c7e9d03589826d477060f06c8c1b7e93" exitCode=0 Jul 14 18:46:27.896801 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:27.896798 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" Jul 14 18:46:27.897013 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:27.896839 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" event={"ID":"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d","Type":"ContainerDied","Data":"5382d7e1fa0fb30ac7fcaa3b0c04e7f8c7e9d03589826d477060f06c8c1b7e93"} Jul 14 18:46:27.897013 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:27.896885 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x" event={"ID":"6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d","Type":"ContainerDied","Data":"71c3fa008f8ad943a383436c9be00e1914d8240cdb80c20d52663a8a2e949fb4"} Jul 14 18:46:27.897013 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:27.896904 2565 scope.go:117] "RemoveContainer" containerID="5382d7e1fa0fb30ac7fcaa3b0c04e7f8c7e9d03589826d477060f06c8c1b7e93" Jul 14 18:46:27.906036 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:27.906020 2565 scope.go:117] "RemoveContainer" containerID="5382d7e1fa0fb30ac7fcaa3b0c04e7f8c7e9d03589826d477060f06c8c1b7e93" Jul 14 18:46:27.906347 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:46:27.906319 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5382d7e1fa0fb30ac7fcaa3b0c04e7f8c7e9d03589826d477060f06c8c1b7e93\": container with ID starting with 5382d7e1fa0fb30ac7fcaa3b0c04e7f8c7e9d03589826d477060f06c8c1b7e93 not found: ID does not exist" containerID="5382d7e1fa0fb30ac7fcaa3b0c04e7f8c7e9d03589826d477060f06c8c1b7e93" Jul 14 18:46:27.906431 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:27.906354 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5382d7e1fa0fb30ac7fcaa3b0c04e7f8c7e9d03589826d477060f06c8c1b7e93"} err="failed to get container status \"5382d7e1fa0fb30ac7fcaa3b0c04e7f8c7e9d03589826d477060f06c8c1b7e93\": rpc error: code = NotFound desc = could not find container \"5382d7e1fa0fb30ac7fcaa3b0c04e7f8c7e9d03589826d477060f06c8c1b7e93\": container with ID starting with 5382d7e1fa0fb30ac7fcaa3b0c04e7f8c7e9d03589826d477060f06c8c1b7e93 not found: ID does not exist" Jul 14 18:46:27.923692 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:27.923665 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x"] Jul 14 18:46:27.930071 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:27.930047 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1f114d/test-model-1f114d-kserve-855c78fd7f-2q55x"] Jul 14 18:46:29.690652 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:29.690610 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d" path="/var/lib/kubelet/pods/6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d/volumes" Jul 14 18:46:34.370294 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:34.370253 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:46:39.033837 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:39.033801 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5"] Jul 14 18:46:39.034336 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:39.034032 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" podUID="c90c4714-014c-47a4-9377-47fc83be7dc5" containerName="istio-proxy" containerID="cri-o://d4260e15e63e2128b42eb4179f1dce7fe0d862907163919c127398c00602a07a" gracePeriod=30 Jul 14 18:46:39.431900 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:39.431810 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7965f3-kn29s"] Jul 14 18:46:39.432253 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:39.432238 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c6ca669-5fc0-403c-8627-10ad92528262" containerName="revoke-keys" Jul 14 18:46:39.432304 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:39.432256 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c6ca669-5fc0-403c-8627-10ad92528262" containerName="revoke-keys" Jul 14 18:46:39.432304 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:39.432273 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d" containerName="main" Jul 14 18:46:39.432304 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:39.432278 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d" containerName="main" Jul 14 18:46:39.432396 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:39.432340 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f3d13dc-b43a-484d-87a2-5f2eab6a6a9d" containerName="main" Jul 14 18:46:39.432396 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:39.432347 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c6ca669-5fc0-403c-8627-10ad92528262" containerName="revoke-keys" Jul 14 18:46:39.435873 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:39.435854 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7965f3-kn29s" Jul 14 18:46:39.438907 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:39.438880 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9jplk\"" Jul 14 18:46:39.439041 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:39.438909 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 18:46:39.448164 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:39.448128 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7965f3-kn29s"] Jul 14 18:46:39.457961 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:39.457922 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53c7b8ef-ff7a-4a99-870b-e0e5104e7258-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7965f3-kn29s\" (UID: \"53c7b8ef-ff7a-4a99-870b-e0e5104e7258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7965f3-kn29s" Jul 14 18:46:39.558478 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:39.558439 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53c7b8ef-ff7a-4a99-870b-e0e5104e7258-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7965f3-kn29s\" (UID: \"53c7b8ef-ff7a-4a99-870b-e0e5104e7258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7965f3-kn29s" Jul 14 18:46:39.559153 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:39.559132 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53c7b8ef-ff7a-4a99-870b-e0e5104e7258-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7965f3-kn29s\" (UID: \"53c7b8ef-ff7a-4a99-870b-e0e5104e7258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7965f3-kn29s" Jul 14 18:46:39.745808 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:39.745720 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7965f3-kn29s" Jul 14 18:46:39.903447 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:39.903412 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7965f3-kn29s"] Jul 14 18:46:39.904628 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:46:39.904602 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c7b8ef_ff7a_4a99_870b_e0e5104e7258.slice/crio-e2656cd7cae452dc18dec56bd945d50a34aa3b4d9b86f6f5164554441e11985d WatchSource:0}: Error finding container e2656cd7cae452dc18dec56bd945d50a34aa3b4d9b86f6f5164554441e11985d: Status 404 returned error can't find the container with id e2656cd7cae452dc18dec56bd945d50a34aa3b4d9b86f6f5164554441e11985d Jul 14 18:46:39.960724 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:39.960685 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7965f3-kn29s" event={"ID":"53c7b8ef-ff7a-4a99-870b-e0e5104e7258","Type":"ContainerStarted","Data":"e2656cd7cae452dc18dec56bd945d50a34aa3b4d9b86f6f5164554441e11985d"} Jul 14 18:46:40.966928 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:40.966832 2565 generic.go:356] "Generic (PLEG): container finished" podID="53c7b8ef-ff7a-4a99-870b-e0e5104e7258" containerID="3f52875b744a8d2284a31f92e15f47d0bb187214736fbd47152a0066e398cdcf" exitCode=0 Jul 14 18:46:40.966928 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:40.966903 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7965f3-kn29s" event={"ID":"53c7b8ef-ff7a-4a99-870b-e0e5104e7258","Type":"ContainerDied","Data":"3f52875b744a8d2284a31f92e15f47d0bb187214736fbd47152a0066e398cdcf"} Jul 14 18:46:42.115545 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:42.115515 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7965f3-kn29s" Jul 14 18:46:42.179285 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:42.179251 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53c7b8ef-ff7a-4a99-870b-e0e5104e7258-maas-api-service-ca\") pod \"53c7b8ef-ff7a-4a99-870b-e0e5104e7258\" (UID: \"53c7b8ef-ff7a-4a99-870b-e0e5104e7258\") " Jul 14 18:46:42.179600 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:42.179575 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53c7b8ef-ff7a-4a99-870b-e0e5104e7258-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "53c7b8ef-ff7a-4a99-870b-e0e5104e7258" (UID: "53c7b8ef-ff7a-4a99-870b-e0e5104e7258"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:46:42.280167 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:42.280136 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53c7b8ef-ff7a-4a99-870b-e0e5104e7258-maas-api-service-ca\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:46:42.978952 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:42.978917 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7965f3-kn29s" Jul 14 18:46:42.979152 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:42.978961 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7965f3-kn29s" event={"ID":"53c7b8ef-ff7a-4a99-870b-e0e5104e7258","Type":"ContainerDied","Data":"e2656cd7cae452dc18dec56bd945d50a34aa3b4d9b86f6f5164554441e11985d"} Jul 14 18:46:42.979152 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:42.979013 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2656cd7cae452dc18dec56bd945d50a34aa3b4d9b86f6f5164554441e11985d" Jul 14 18:46:44.298219 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.298155 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:46:44.395485 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.395442 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c90c4714-014c-47a4-9377-47fc83be7dc5-istio-token\") pod \"c90c4714-014c-47a4-9377-47fc83be7dc5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " Jul 14 18:46:44.395696 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.395496 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c90c4714-014c-47a4-9377-47fc83be7dc5-istio-podinfo\") pod \"c90c4714-014c-47a4-9377-47fc83be7dc5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " Jul 14 18:46:44.395696 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.395524 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c90c4714-014c-47a4-9377-47fc83be7dc5-istio-data\") pod \"c90c4714-014c-47a4-9377-47fc83be7dc5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " Jul 14 18:46:44.395696 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.395567 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c90c4714-014c-47a4-9377-47fc83be7dc5-workload-socket\") pod \"c90c4714-014c-47a4-9377-47fc83be7dc5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " Jul 14 18:46:44.395696 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.395591 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c90c4714-014c-47a4-9377-47fc83be7dc5-istio-envoy\") pod \"c90c4714-014c-47a4-9377-47fc83be7dc5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " Jul 14 18:46:44.395926 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.395746 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c90c4714-014c-47a4-9377-47fc83be7dc5-credential-socket\") pod \"c90c4714-014c-47a4-9377-47fc83be7dc5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " Jul 14 18:46:44.395926 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.395827 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c90c4714-014c-47a4-9377-47fc83be7dc5-istiod-ca-cert\") pod \"c90c4714-014c-47a4-9377-47fc83be7dc5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " Jul 14 18:46:44.395926 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.395872 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c90c4714-014c-47a4-9377-47fc83be7dc5-workload-certs\") pod \"c90c4714-014c-47a4-9377-47fc83be7dc5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " Jul 14 18:46:44.396079 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.395933 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c90c4714-014c-47a4-9377-47fc83be7dc5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c90c4714-014c-47a4-9377-47fc83be7dc5" (UID: "c90c4714-014c-47a4-9377-47fc83be7dc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:46:44.396079 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.395946 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4x8n\" (UniqueName: \"kubernetes.io/projected/c90c4714-014c-47a4-9377-47fc83be7dc5-kube-api-access-f4x8n\") pod \"c90c4714-014c-47a4-9377-47fc83be7dc5\" (UID: \"c90c4714-014c-47a4-9377-47fc83be7dc5\") " Jul 14 18:46:44.396405 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.396321 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c90c4714-014c-47a4-9377-47fc83be7dc5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c90c4714-014c-47a4-9377-47fc83be7dc5" (UID: "c90c4714-014c-47a4-9377-47fc83be7dc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:46:44.396616 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.396495 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c90c4714-014c-47a4-9377-47fc83be7dc5-istiod-ca-cert\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:46:44.396616 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.396517 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c90c4714-014c-47a4-9377-47fc83be7dc5-workload-socket\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:46:44.396736 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.396620 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c90c4714-014c-47a4-9377-47fc83be7dc5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c90c4714-014c-47a4-9377-47fc83be7dc5" (UID: "c90c4714-014c-47a4-9377-47fc83be7dc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:46:44.396736 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.396714 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c90c4714-014c-47a4-9377-47fc83be7dc5-istio-data" (OuterVolumeSpecName: "istio-data") pod "c90c4714-014c-47a4-9377-47fc83be7dc5" (UID: "c90c4714-014c-47a4-9377-47fc83be7dc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:46:44.396854 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.396733 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c90c4714-014c-47a4-9377-47fc83be7dc5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c90c4714-014c-47a4-9377-47fc83be7dc5" (UID: "c90c4714-014c-47a4-9377-47fc83be7dc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:46:44.397993 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.397967 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c90c4714-014c-47a4-9377-47fc83be7dc5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c90c4714-014c-47a4-9377-47fc83be7dc5" (UID: "c90c4714-014c-47a4-9377-47fc83be7dc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:46:44.398077 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.397996 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c90c4714-014c-47a4-9377-47fc83be7dc5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c90c4714-014c-47a4-9377-47fc83be7dc5" (UID: "c90c4714-014c-47a4-9377-47fc83be7dc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 18:46:44.398077 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.398005 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c90c4714-014c-47a4-9377-47fc83be7dc5-istio-token" (OuterVolumeSpecName: "istio-token") pod "c90c4714-014c-47a4-9377-47fc83be7dc5" (UID: "c90c4714-014c-47a4-9377-47fc83be7dc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:46:44.398445 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.398427 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c90c4714-014c-47a4-9377-47fc83be7dc5-kube-api-access-f4x8n" (OuterVolumeSpecName: "kube-api-access-f4x8n") pod "c90c4714-014c-47a4-9377-47fc83be7dc5" (UID: "c90c4714-014c-47a4-9377-47fc83be7dc5"). InnerVolumeSpecName "kube-api-access-f4x8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:46:44.497130 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.497063 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c90c4714-014c-47a4-9377-47fc83be7dc5-credential-socket\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:46:44.497130 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.497123 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c90c4714-014c-47a4-9377-47fc83be7dc5-workload-certs\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:46:44.497130 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.497136 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4x8n\" (UniqueName: \"kubernetes.io/projected/c90c4714-014c-47a4-9377-47fc83be7dc5-kube-api-access-f4x8n\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:46:44.497381 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.497148 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c90c4714-014c-47a4-9377-47fc83be7dc5-istio-token\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:46:44.497381 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.497159 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c90c4714-014c-47a4-9377-47fc83be7dc5-istio-podinfo\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:46:44.497381 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.497169 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c90c4714-014c-47a4-9377-47fc83be7dc5-istio-data\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:46:44.497381 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.497178 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c90c4714-014c-47a4-9377-47fc83be7dc5-istio-envoy\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:46:44.991117 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.991067 2565 generic.go:356] "Generic (PLEG): container finished" podID="c90c4714-014c-47a4-9377-47fc83be7dc5" containerID="d4260e15e63e2128b42eb4179f1dce7fe0d862907163919c127398c00602a07a" exitCode=0 Jul 14 18:46:44.991271 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.991145 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" Jul 14 18:46:44.991271 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.991142 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" event={"ID":"c90c4714-014c-47a4-9377-47fc83be7dc5","Type":"ContainerDied","Data":"d4260e15e63e2128b42eb4179f1dce7fe0d862907163919c127398c00602a07a"} Jul 14 18:46:44.991370 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.991279 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5" event={"ID":"c90c4714-014c-47a4-9377-47fc83be7dc5","Type":"ContainerDied","Data":"a431824c932034ab4a33e29c2b20402518e2d631cbab1efc04a756f3021bc2c7"} Jul 14 18:46:44.991370 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:44.991309 2565 scope.go:117] "RemoveContainer" containerID="d4260e15e63e2128b42eb4179f1dce7fe0d862907163919c127398c00602a07a" Jul 14 18:46:45.001170 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:45.001149 2565 scope.go:117] "RemoveContainer" containerID="5d6b68de9262bc318fcff9d7f91db7e7d89bdc94839d35eaa63d35d840e45ffc" Jul 14 18:46:45.009531 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:45.009511 2565 scope.go:117] "RemoveContainer" containerID="d4260e15e63e2128b42eb4179f1dce7fe0d862907163919c127398c00602a07a" Jul 14 18:46:45.009797 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:46:45.009778 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4260e15e63e2128b42eb4179f1dce7fe0d862907163919c127398c00602a07a\": container with ID starting with d4260e15e63e2128b42eb4179f1dce7fe0d862907163919c127398c00602a07a not found: ID does not exist" containerID="d4260e15e63e2128b42eb4179f1dce7fe0d862907163919c127398c00602a07a" Jul 14 18:46:45.009898 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:45.009807 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4260e15e63e2128b42eb4179f1dce7fe0d862907163919c127398c00602a07a"} err="failed to get container status \"d4260e15e63e2128b42eb4179f1dce7fe0d862907163919c127398c00602a07a\": rpc error: code = NotFound desc = could not find container \"d4260e15e63e2128b42eb4179f1dce7fe0d862907163919c127398c00602a07a\": container with ID starting with d4260e15e63e2128b42eb4179f1dce7fe0d862907163919c127398c00602a07a not found: ID does not exist" Jul 14 18:46:45.009898 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:45.009825 2565 scope.go:117] "RemoveContainer" containerID="5d6b68de9262bc318fcff9d7f91db7e7d89bdc94839d35eaa63d35d840e45ffc" Jul 14 18:46:45.010016 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:46:45.010001 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d6b68de9262bc318fcff9d7f91db7e7d89bdc94839d35eaa63d35d840e45ffc\": container with ID starting with 5d6b68de9262bc318fcff9d7f91db7e7d89bdc94839d35eaa63d35d840e45ffc not found: ID does not exist" containerID="5d6b68de9262bc318fcff9d7f91db7e7d89bdc94839d35eaa63d35d840e45ffc" Jul 14 18:46:45.010052 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:45.010021 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d6b68de9262bc318fcff9d7f91db7e7d89bdc94839d35eaa63d35d840e45ffc"} err="failed to get container status \"5d6b68de9262bc318fcff9d7f91db7e7d89bdc94839d35eaa63d35d840e45ffc\": rpc error: code = NotFound desc = could not find container \"5d6b68de9262bc318fcff9d7f91db7e7d89bdc94839d35eaa63d35d840e45ffc\": container with ID starting with 5d6b68de9262bc318fcff9d7f91db7e7d89bdc94839d35eaa63d35d840e45ffc not found: ID does not exist" Jul 14 18:46:45.015298 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:45.015272 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5"] Jul 14 18:46:45.019821 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:45.019800 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1f114d-openshift-default-6b4f45b47-wv9z5"] Jul 14 18:46:45.692228 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:45.692198 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c90c4714-014c-47a4-9377-47fc83be7dc5" path="/var/lib/kubelet/pods/c90c4714-014c-47a4-9377-47fc83be7dc5/volumes" Jul 14 18:46:50.023893 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:50.023857 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7965f3-29734245-4n7b8"] Jul 14 18:46:50.027264 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:50.027237 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7965f3-29734245-4n7b8"] Jul 14 18:46:51.690149 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:51.690087 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec5fce56-a1be-44eb-a58c-91f1d7eaff5a" path="/var/lib/kubelet/pods/ec5fce56-a1be-44eb-a58c-91f1d7eaff5a/volumes" Jul 14 18:46:53.157758 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:46:53.157719 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:47:06.297672 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.297630 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v"] Jul 14 18:47:06.298248 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.297968 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53c7b8ef-ff7a-4a99-870b-e0e5104e7258" containerName="revoke-keys" Jul 14 18:47:06.298248 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.297979 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="53c7b8ef-ff7a-4a99-870b-e0e5104e7258" containerName="revoke-keys" Jul 14 18:47:06.298248 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.297991 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c90c4714-014c-47a4-9377-47fc83be7dc5" containerName="istio-proxy" Jul 14 18:47:06.298248 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.297997 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c90c4714-014c-47a4-9377-47fc83be7dc5" containerName="istio-proxy" Jul 14 18:47:06.298248 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.298005 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c90c4714-014c-47a4-9377-47fc83be7dc5" containerName="istio-proxy" Jul 14 18:47:06.298248 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.298010 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c90c4714-014c-47a4-9377-47fc83be7dc5" containerName="istio-proxy" Jul 14 18:47:06.298248 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.298072 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c90c4714-014c-47a4-9377-47fc83be7dc5" containerName="istio-proxy" Jul 14 18:47:06.298248 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.298080 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c90c4714-014c-47a4-9377-47fc83be7dc5" containerName="istio-proxy" Jul 14 18:47:06.298248 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.298085 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="53c7b8ef-ff7a-4a99-870b-e0e5104e7258" containerName="revoke-keys" Jul 14 18:47:06.301426 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.301404 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:06.304180 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.304157 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-fade18-openshift-default-dockercfg-5fvl7\"" Jul 14 18:47:06.319017 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.318994 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v"] Jul 14 18:47:06.388488 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.388452 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-istio-token\") pod \"e2e-api-a-fade18-openshift-default-575f9899fc-tql6v\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:06.388757 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.388737 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-istio-envoy\") pod \"e2e-api-a-fade18-openshift-default-575f9899fc-tql6v\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:06.388911 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.388887 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-workload-certs\") pod \"e2e-api-a-fade18-openshift-default-575f9899fc-tql6v\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:06.389047 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.388919 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-istio-podinfo\") pod \"e2e-api-a-fade18-openshift-default-575f9899fc-tql6v\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:06.389047 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.388965 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-istio-data\") pod \"e2e-api-a-fade18-openshift-default-575f9899fc-tql6v\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:06.389188 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.389050 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-credential-socket\") pod \"e2e-api-a-fade18-openshift-default-575f9899fc-tql6v\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:06.389188 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.389118 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-istiod-ca-cert\") pod \"e2e-api-a-fade18-openshift-default-575f9899fc-tql6v\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:06.389261 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.389213 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7st7v\" (UniqueName: \"kubernetes.io/projected/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-kube-api-access-7st7v\") pod \"e2e-api-a-fade18-openshift-default-575f9899fc-tql6v\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:06.389261 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.389249 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-workload-socket\") pod \"e2e-api-a-fade18-openshift-default-575f9899fc-tql6v\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:06.489865 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.489834 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-istio-token\") pod \"e2e-api-a-fade18-openshift-default-575f9899fc-tql6v\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:06.490073 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.489875 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-istio-envoy\") pod \"e2e-api-a-fade18-openshift-default-575f9899fc-tql6v\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:06.490073 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.489908 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-workload-certs\") pod \"e2e-api-a-fade18-openshift-default-575f9899fc-tql6v\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:06.490073 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.489942 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-istio-podinfo\") pod \"e2e-api-a-fade18-openshift-default-575f9899fc-tql6v\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:06.490073 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.489981 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-istio-data\") pod \"e2e-api-a-fade18-openshift-default-575f9899fc-tql6v\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:06.490073 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.490022 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-credential-socket\") pod \"e2e-api-a-fade18-openshift-default-575f9899fc-tql6v\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:06.490073 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.490047 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-istiod-ca-cert\") pod \"e2e-api-a-fade18-openshift-default-575f9899fc-tql6v\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:06.490437 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.490131 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7st7v\" (UniqueName: \"kubernetes.io/projected/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-kube-api-access-7st7v\") pod \"e2e-api-a-fade18-openshift-default-575f9899fc-tql6v\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:06.490437 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.490157 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-workload-socket\") pod \"e2e-api-a-fade18-openshift-default-575f9899fc-tql6v\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:06.490437 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.490333 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-workload-certs\") pod \"e2e-api-a-fade18-openshift-default-575f9899fc-tql6v\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:06.490600 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.490442 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-workload-socket\") pod \"e2e-api-a-fade18-openshift-default-575f9899fc-tql6v\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:06.491037 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.490990 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-istiod-ca-cert\") pod \"e2e-api-a-fade18-openshift-default-575f9899fc-tql6v\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:06.491743 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.491699 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-istio-data\") pod \"e2e-api-a-fade18-openshift-default-575f9899fc-tql6v\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:06.492073 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.492055 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-credential-socket\") pod \"e2e-api-a-fade18-openshift-default-575f9899fc-tql6v\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:06.492935 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.492905 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-istio-envoy\") pod \"e2e-api-a-fade18-openshift-default-575f9899fc-tql6v\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:06.494434 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.494400 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-istio-podinfo\") pod \"e2e-api-a-fade18-openshift-default-575f9899fc-tql6v\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:06.500051 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.500028 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-istio-token\") pod \"e2e-api-a-fade18-openshift-default-575f9899fc-tql6v\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:06.500552 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.500535 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7st7v\" (UniqueName: \"kubernetes.io/projected/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-kube-api-access-7st7v\") pod \"e2e-api-a-fade18-openshift-default-575f9899fc-tql6v\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:06.611494 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.611412 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:06.764798 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.764776 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v"] Jul 14 18:47:06.766694 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:47:06.766651 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda20bdd60_1bd8_413b_87aa_29a19a23d6b6.slice/crio-9f304a403641435738fa306deeb832c771e5ebd02a8bd6aa0a8aef4b690e4255 WatchSource:0}: Error finding container 9f304a403641435738fa306deeb832c771e5ebd02a8bd6aa0a8aef4b690e4255: Status 404 returned error can't find the container with id 9f304a403641435738fa306deeb832c771e5ebd02a8bd6aa0a8aef4b690e4255 Jul 14 18:47:06.769023 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.768993 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:47:06.769139 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.769075 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:47:06.769139 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:06.769119 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:47:07.108738 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:07.108699 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" event={"ID":"a20bdd60-1bd8-413b-87aa-29a19a23d6b6","Type":"ContainerStarted","Data":"25c74ef842da11e16d505e50f668c8750280cb35d23adc2029fb97d0a4d187a3"} Jul 14 18:47:07.108905 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:07.108757 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" event={"ID":"a20bdd60-1bd8-413b-87aa-29a19a23d6b6","Type":"ContainerStarted","Data":"9f304a403641435738fa306deeb832c771e5ebd02a8bd6aa0a8aef4b690e4255"} Jul 14 18:47:07.131029 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:07.130983 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" podStartSLOduration=1.130969226 podStartE2EDuration="1.130969226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:47:07.12842755 +0000 UTC m=+2231.965770806" watchObservedRunningTime="2026-07-14 18:47:07.130969226 +0000 UTC m=+2231.968312483" Jul 14 18:47:07.612441 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:07.612407 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:07.617316 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:07.617292 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:08.113868 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:08.113832 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:08.114940 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:08.114918 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:13.858792 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:13.858751 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:47:23.115157 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:23.115030 2565 patch_prober.go:28] interesting pod/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 14 18:47:23.115157 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:23.115124 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" podUID="a20bdd60-1bd8-413b-87aa-29a19a23d6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 14 18:47:23.192808 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:23.192777 2565 generic.go:356] "Generic (PLEG): container finished" podID="a20bdd60-1bd8-413b-87aa-29a19a23d6b6" containerID="25c74ef842da11e16d505e50f668c8750280cb35d23adc2029fb97d0a4d187a3" exitCode=0 Jul 14 18:47:23.192988 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:23.192849 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" event={"ID":"a20bdd60-1bd8-413b-87aa-29a19a23d6b6","Type":"ContainerDied","Data":"25c74ef842da11e16d505e50f668c8750280cb35d23adc2029fb97d0a4d187a3"} Jul 14 18:47:23.193283 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:23.193269 2565 scope.go:117] "RemoveContainer" containerID="25c74ef842da11e16d505e50f668c8750280cb35d23adc2029fb97d0a4d187a3" Jul 14 18:47:23.194335 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:23.194308 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:47:23.194462 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:23.194392 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:47:23.194462 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:23.194441 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:47:24.203146 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:24.203074 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" event={"ID":"a20bdd60-1bd8-413b-87aa-29a19a23d6b6","Type":"ContainerStarted","Data":"62dfe7298b8271b90ef0c5593ce20e567d7076cbb24b1156b2ccf9cefc30aef2"} Jul 14 18:47:24.611795 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:24.611750 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:24.612235 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:24.612206 2565 patch_prober.go:28] interesting pod/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 14 18:47:24.612367 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:24.612271 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" podUID="a20bdd60-1bd8-413b-87aa-29a19a23d6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 14 18:47:25.612119 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:25.612052 2565 patch_prober.go:28] interesting pod/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 14 18:47:25.612561 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:25.612144 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" podUID="a20bdd60-1bd8-413b-87aa-29a19a23d6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 14 18:47:26.612922 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:26.612869 2565 patch_prober.go:28] interesting pod/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 14 18:47:26.613359 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:26.612953 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" podUID="a20bdd60-1bd8-413b-87aa-29a19a23d6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 14 18:47:27.612211 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:27.612169 2565 patch_prober.go:28] interesting pod/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 14 18:47:27.612424 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:27.612240 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" podUID="a20bdd60-1bd8-413b-87aa-29a19a23d6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 14 18:47:28.592719 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:28.592675 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:47:28.615920 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:28.615892 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:28.616168 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:28.616140 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:28.616725 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:28.616708 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:47:30.608924 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:30.608887 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fade18-bbvlm"] Jul 14 18:47:30.612585 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:30.612560 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fade18-bbvlm" Jul 14 18:47:30.615223 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:30.615202 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 18:47:30.615359 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:30.615206 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9jplk\"" Jul 14 18:47:30.620755 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:30.620725 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fade18-bbvlm"] Jul 14 18:47:30.621055 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:30.621027 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f52e563-9f2b-4a80-bced-88c919bc3e7b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fade18-bbvlm\" (UID: \"1f52e563-9f2b-4a80-bced-88c919bc3e7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fade18-bbvlm" Jul 14 18:47:30.722592 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:30.722557 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f52e563-9f2b-4a80-bced-88c919bc3e7b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fade18-bbvlm\" (UID: \"1f52e563-9f2b-4a80-bced-88c919bc3e7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fade18-bbvlm" Jul 14 18:47:30.723317 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:30.723295 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f52e563-9f2b-4a80-bced-88c919bc3e7b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fade18-bbvlm\" (UID: \"1f52e563-9f2b-4a80-bced-88c919bc3e7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fade18-bbvlm" Jul 14 18:47:30.924656 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:30.924557 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fade18-bbvlm" Jul 14 18:47:31.059044 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:31.059008 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fade18-bbvlm"] Jul 14 18:47:31.060947 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:47:31.060921 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f52e563_9f2b_4a80_bced_88c919bc3e7b.slice/crio-6cfaa78c1b498ff1ee32caf0389c0c025afe90fa5fbd6615f00d99713e88a7ca WatchSource:0}: Error finding container 6cfaa78c1b498ff1ee32caf0389c0c025afe90fa5fbd6615f00d99713e88a7ca: Status 404 returned error can't find the container with id 6cfaa78c1b498ff1ee32caf0389c0c025afe90fa5fbd6615f00d99713e88a7ca Jul 14 18:47:31.244816 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:31.244730 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fade18-bbvlm" event={"ID":"1f52e563-9f2b-4a80-bced-88c919bc3e7b","Type":"ContainerStarted","Data":"6cfaa78c1b498ff1ee32caf0389c0c025afe90fa5fbd6615f00d99713e88a7ca"} Jul 14 18:47:32.251248 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:32.251148 2565 generic.go:356] "Generic (PLEG): container finished" podID="1f52e563-9f2b-4a80-bced-88c919bc3e7b" containerID="16c89906812a768eaa883634ea0068cc9f7c4fa4238da05538034bfb9729c937" exitCode=0 Jul 14 18:47:32.251248 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:32.251193 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fade18-bbvlm" event={"ID":"1f52e563-9f2b-4a80-bced-88c919bc3e7b","Type":"ContainerDied","Data":"16c89906812a768eaa883634ea0068cc9f7c4fa4238da05538034bfb9729c937"} Jul 14 18:47:33.393660 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:33.393623 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fade18-bbvlm" Jul 14 18:47:33.447968 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:33.447933 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f52e563-9f2b-4a80-bced-88c919bc3e7b-maas-api-service-ca\") pod \"1f52e563-9f2b-4a80-bced-88c919bc3e7b\" (UID: \"1f52e563-9f2b-4a80-bced-88c919bc3e7b\") " Jul 14 18:47:33.448288 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:33.448267 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f52e563-9f2b-4a80-bced-88c919bc3e7b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1f52e563-9f2b-4a80-bced-88c919bc3e7b" (UID: "1f52e563-9f2b-4a80-bced-88c919bc3e7b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:47:33.548974 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:33.548943 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f52e563-9f2b-4a80-bced-88c919bc3e7b-maas-api-service-ca\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:47:34.262414 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:34.262383 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fade18-bbvlm" Jul 14 18:47:34.262414 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:34.262397 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fade18-bbvlm" event={"ID":"1f52e563-9f2b-4a80-bced-88c919bc3e7b","Type":"ContainerDied","Data":"6cfaa78c1b498ff1ee32caf0389c0c025afe90fa5fbd6615f00d99713e88a7ca"} Jul 14 18:47:34.262631 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:34.262446 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cfaa78c1b498ff1ee32caf0389c0c025afe90fa5fbd6615f00d99713e88a7ca" Jul 14 18:47:45.047936 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:45.047897 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:47:56.436389 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:56.436358 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v"] Jul 14 18:47:56.436812 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:56.436567 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" podUID="a20bdd60-1bd8-413b-87aa-29a19a23d6b6" containerName="istio-proxy" containerID="cri-o://62dfe7298b8271b90ef0c5593ce20e567d7076cbb24b1156b2ccf9cefc30aef2" gracePeriod=30 Jul 14 18:47:56.844626 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:56.844591 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fde6bc-4dw7l"] Jul 14 18:47:56.844943 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:56.844931 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f52e563-9f2b-4a80-bced-88c919bc3e7b" containerName="revoke-keys" Jul 14 18:47:56.844985 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:56.844945 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f52e563-9f2b-4a80-bced-88c919bc3e7b" containerName="revoke-keys" Jul 14 18:47:56.845037 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:56.845026 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f52e563-9f2b-4a80-bced-88c919bc3e7b" containerName="revoke-keys" Jul 14 18:47:56.849328 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:56.849312 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fde6bc-4dw7l" Jul 14 18:47:56.851984 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:56.851964 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9jplk\"" Jul 14 18:47:56.851984 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:56.851975 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 18:47:56.857038 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:56.856886 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fde6bc-4dw7l"] Jul 14 18:47:56.954550 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:56.954511 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5dfc7bc-5a1d-4753-9cb9-286c59ddc6eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-fde6bc-4dw7l\" (UID: \"f5dfc7bc-5a1d-4753-9cb9-286c59ddc6eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fde6bc-4dw7l" Jul 14 18:47:57.055678 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:57.055640 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5dfc7bc-5a1d-4753-9cb9-286c59ddc6eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-fde6bc-4dw7l\" (UID: \"f5dfc7bc-5a1d-4753-9cb9-286c59ddc6eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fde6bc-4dw7l" Jul 14 18:47:57.056290 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:57.056270 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5dfc7bc-5a1d-4753-9cb9-286c59ddc6eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-fde6bc-4dw7l\" (UID: \"f5dfc7bc-5a1d-4753-9cb9-286c59ddc6eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fde6bc-4dw7l" Jul 14 18:47:57.160241 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:57.160153 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fde6bc-4dw7l" Jul 14 18:47:57.293316 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:57.293288 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fde6bc-4dw7l"] Jul 14 18:47:57.295788 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:47:57.295761 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5dfc7bc_5a1d_4753_9cb9_286c59ddc6eb.slice/crio-c655e44ab3c30b878ca167fd54157762786f1abe698629230f6ba383a820493a WatchSource:0}: Error finding container c655e44ab3c30b878ca167fd54157762786f1abe698629230f6ba383a820493a: Status 404 returned error can't find the container with id c655e44ab3c30b878ca167fd54157762786f1abe698629230f6ba383a820493a Jul 14 18:47:57.380930 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:57.380898 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fde6bc-4dw7l" event={"ID":"f5dfc7bc-5a1d-4753-9cb9-286c59ddc6eb","Type":"ContainerStarted","Data":"c655e44ab3c30b878ca167fd54157762786f1abe698629230f6ba383a820493a"} Jul 14 18:47:58.388221 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:58.388088 2565 generic.go:356] "Generic (PLEG): container finished" podID="f5dfc7bc-5a1d-4753-9cb9-286c59ddc6eb" containerID="99c6b75cc61a424210bda808d005a5a13684d3f7454aea1f28e065431771b364" exitCode=0 Jul 14 18:47:58.388221 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:58.388170 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fde6bc-4dw7l" event={"ID":"f5dfc7bc-5a1d-4753-9cb9-286c59ddc6eb","Type":"ContainerDied","Data":"99c6b75cc61a424210bda808d005a5a13684d3f7454aea1f28e065431771b364"} Jul 14 18:47:58.618964 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:58.618930 2565 patch_prober.go:28] interesting pod/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 18:47:58.619132 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:58.618993 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" podUID="a20bdd60-1bd8-413b-87aa-29a19a23d6b6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 18:47:59.526211 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:59.526184 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fde6bc-4dw7l" Jul 14 18:47:59.679008 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:59.678920 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5dfc7bc-5a1d-4753-9cb9-286c59ddc6eb-maas-api-service-ca\") pod \"f5dfc7bc-5a1d-4753-9cb9-286c59ddc6eb\" (UID: \"f5dfc7bc-5a1d-4753-9cb9-286c59ddc6eb\") " Jul 14 18:47:59.679313 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:59.679290 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5dfc7bc-5a1d-4753-9cb9-286c59ddc6eb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f5dfc7bc-5a1d-4753-9cb9-286c59ddc6eb" (UID: "f5dfc7bc-5a1d-4753-9cb9-286c59ddc6eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:47:59.779763 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:47:59.779734 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5dfc7bc-5a1d-4753-9cb9-286c59ddc6eb-maas-api-service-ca\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:48:00.400642 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:00.400610 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fde6bc-4dw7l" Jul 14 18:48:00.400642 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:00.400622 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fde6bc-4dw7l" event={"ID":"f5dfc7bc-5a1d-4753-9cb9-286c59ddc6eb","Type":"ContainerDied","Data":"c655e44ab3c30b878ca167fd54157762786f1abe698629230f6ba383a820493a"} Jul 14 18:48:00.400866 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:00.400673 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c655e44ab3c30b878ca167fd54157762786f1abe698629230f6ba383a820493a" Jul 14 18:48:01.697707 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:01.697636 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:48:01.799301 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:01.799266 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7st7v\" (UniqueName: \"kubernetes.io/projected/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-kube-api-access-7st7v\") pod \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " Jul 14 18:48:01.799509 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:01.799317 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-istio-envoy\") pod \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " Jul 14 18:48:01.799509 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:01.799361 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-credential-socket\") pod \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " Jul 14 18:48:01.799509 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:01.799405 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-workload-certs\") pod \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " Jul 14 18:48:01.799509 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:01.799439 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-istio-token\") pod \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " Jul 14 18:48:01.799509 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:01.799464 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-workload-socket\") pod \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " Jul 14 18:48:01.799509 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:01.799506 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-istio-podinfo\") pod \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " Jul 14 18:48:01.799852 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:01.799549 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-istiod-ca-cert\") pod \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " Jul 14 18:48:01.799852 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:01.799591 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-istio-data\") pod \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\" (UID: \"a20bdd60-1bd8-413b-87aa-29a19a23d6b6\") " Jul 14 18:48:01.799852 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:01.799664 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a20bdd60-1bd8-413b-87aa-29a19a23d6b6" (UID: "a20bdd60-1bd8-413b-87aa-29a19a23d6b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:48:01.799998 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:01.799979 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a20bdd60-1bd8-413b-87aa-29a19a23d6b6" (UID: "a20bdd60-1bd8-413b-87aa-29a19a23d6b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:48:01.800692 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:01.800175 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-credential-socket\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:48:01.800692 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:01.800508 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a20bdd60-1bd8-413b-87aa-29a19a23d6b6" (UID: "a20bdd60-1bd8-413b-87aa-29a19a23d6b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:48:01.800941 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:01.800911 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-istio-data" (OuterVolumeSpecName: "istio-data") pod "a20bdd60-1bd8-413b-87aa-29a19a23d6b6" (UID: "a20bdd60-1bd8-413b-87aa-29a19a23d6b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:48:01.801065 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:01.800920 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a20bdd60-1bd8-413b-87aa-29a19a23d6b6" (UID: "a20bdd60-1bd8-413b-87aa-29a19a23d6b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:48:01.801950 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:01.801912 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-kube-api-access-7st7v" (OuterVolumeSpecName: "kube-api-access-7st7v") pod "a20bdd60-1bd8-413b-87aa-29a19a23d6b6" (UID: "a20bdd60-1bd8-413b-87aa-29a19a23d6b6"). InnerVolumeSpecName "kube-api-access-7st7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:48:01.802091 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:01.802066 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a20bdd60-1bd8-413b-87aa-29a19a23d6b6" (UID: "a20bdd60-1bd8-413b-87aa-29a19a23d6b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:48:01.802217 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:01.802192 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a20bdd60-1bd8-413b-87aa-29a19a23d6b6" (UID: "a20bdd60-1bd8-413b-87aa-29a19a23d6b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 18:48:01.802569 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:01.802545 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-istio-token" (OuterVolumeSpecName: "istio-token") pod "a20bdd60-1bd8-413b-87aa-29a19a23d6b6" (UID: "a20bdd60-1bd8-413b-87aa-29a19a23d6b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:48:01.901079 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:01.901051 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-workload-certs\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:48:01.901079 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:01.901076 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-istio-token\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:48:01.901079 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:01.901086 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-workload-socket\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:48:01.901289 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:01.901105 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-istio-podinfo\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:48:01.901289 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:01.901115 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-istiod-ca-cert\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:48:01.901289 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:01.901122 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-istio-data\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:48:01.901289 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:01.901130 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7st7v\" (UniqueName: \"kubernetes.io/projected/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-kube-api-access-7st7v\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:48:01.901289 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:01.901138 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a20bdd60-1bd8-413b-87aa-29a19a23d6b6-istio-envoy\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:48:02.412417 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:02.412384 2565 generic.go:356] "Generic (PLEG): container finished" podID="a20bdd60-1bd8-413b-87aa-29a19a23d6b6" containerID="62dfe7298b8271b90ef0c5593ce20e567d7076cbb24b1156b2ccf9cefc30aef2" exitCode=0 Jul 14 18:48:02.412585 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:02.412431 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" Jul 14 18:48:02.412585 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:02.412480 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" event={"ID":"a20bdd60-1bd8-413b-87aa-29a19a23d6b6","Type":"ContainerDied","Data":"62dfe7298b8271b90ef0c5593ce20e567d7076cbb24b1156b2ccf9cefc30aef2"} Jul 14 18:48:02.412585 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:02.412535 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v" event={"ID":"a20bdd60-1bd8-413b-87aa-29a19a23d6b6","Type":"ContainerDied","Data":"9f304a403641435738fa306deeb832c771e5ebd02a8bd6aa0a8aef4b690e4255"} Jul 14 18:48:02.412585 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:02.412560 2565 scope.go:117] "RemoveContainer" containerID="62dfe7298b8271b90ef0c5593ce20e567d7076cbb24b1156b2ccf9cefc30aef2" Jul 14 18:48:02.425402 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:02.425384 2565 scope.go:117] "RemoveContainer" containerID="25c74ef842da11e16d505e50f668c8750280cb35d23adc2029fb97d0a4d187a3" Jul 14 18:48:02.433543 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:02.433523 2565 scope.go:117] "RemoveContainer" containerID="62dfe7298b8271b90ef0c5593ce20e567d7076cbb24b1156b2ccf9cefc30aef2" Jul 14 18:48:02.433775 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:48:02.433755 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62dfe7298b8271b90ef0c5593ce20e567d7076cbb24b1156b2ccf9cefc30aef2\": container with ID starting with 62dfe7298b8271b90ef0c5593ce20e567d7076cbb24b1156b2ccf9cefc30aef2 not found: ID does not exist" containerID="62dfe7298b8271b90ef0c5593ce20e567d7076cbb24b1156b2ccf9cefc30aef2" Jul 14 18:48:02.433846 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:02.433786 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62dfe7298b8271b90ef0c5593ce20e567d7076cbb24b1156b2ccf9cefc30aef2"} err="failed to get container status \"62dfe7298b8271b90ef0c5593ce20e567d7076cbb24b1156b2ccf9cefc30aef2\": rpc error: code = NotFound desc = could not find container \"62dfe7298b8271b90ef0c5593ce20e567d7076cbb24b1156b2ccf9cefc30aef2\": container with ID starting with 62dfe7298b8271b90ef0c5593ce20e567d7076cbb24b1156b2ccf9cefc30aef2 not found: ID does not exist" Jul 14 18:48:02.433846 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:02.433810 2565 scope.go:117] "RemoveContainer" containerID="25c74ef842da11e16d505e50f668c8750280cb35d23adc2029fb97d0a4d187a3" Jul 14 18:48:02.434066 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:48:02.434041 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25c74ef842da11e16d505e50f668c8750280cb35d23adc2029fb97d0a4d187a3\": container with ID starting with 25c74ef842da11e16d505e50f668c8750280cb35d23adc2029fb97d0a4d187a3 not found: ID does not exist" containerID="25c74ef842da11e16d505e50f668c8750280cb35d23adc2029fb97d0a4d187a3" Jul 14 18:48:02.434168 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:02.434074 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25c74ef842da11e16d505e50f668c8750280cb35d23adc2029fb97d0a4d187a3"} err="failed to get container status \"25c74ef842da11e16d505e50f668c8750280cb35d23adc2029fb97d0a4d187a3\": rpc error: code = NotFound desc = could not find container \"25c74ef842da11e16d505e50f668c8750280cb35d23adc2029fb97d0a4d187a3\": container with ID starting with 25c74ef842da11e16d505e50f668c8750280cb35d23adc2029fb97d0a4d187a3 not found: ID does not exist" Jul 14 18:48:02.436589 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:02.436571 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v"] Jul 14 18:48:02.441692 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:02.441663 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-fade18-openshift-default-575f9899fc-tql6v"] Jul 14 18:48:03.689375 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:03.689345 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a20bdd60-1bd8-413b-87aa-29a19a23d6b6" path="/var/lib/kubelet/pods/a20bdd60-1bd8-413b-87aa-29a19a23d6b6/volumes" Jul 14 18:48:10.446844 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:10.446807 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:48:23.737484 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.737451 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr"] Jul 14 18:48:23.738056 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.737915 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5dfc7bc-5a1d-4753-9cb9-286c59ddc6eb" containerName="revoke-keys" Jul 14 18:48:23.738056 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.737933 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5dfc7bc-5a1d-4753-9cb9-286c59ddc6eb" containerName="revoke-keys" Jul 14 18:48:23.738056 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.737945 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a20bdd60-1bd8-413b-87aa-29a19a23d6b6" containerName="istio-proxy" Jul 14 18:48:23.738056 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.737951 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a20bdd60-1bd8-413b-87aa-29a19a23d6b6" containerName="istio-proxy" Jul 14 18:48:23.738056 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.737962 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a20bdd60-1bd8-413b-87aa-29a19a23d6b6" containerName="istio-proxy" Jul 14 18:48:23.738056 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.737967 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a20bdd60-1bd8-413b-87aa-29a19a23d6b6" containerName="istio-proxy" Jul 14 18:48:23.738056 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.738035 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a20bdd60-1bd8-413b-87aa-29a19a23d6b6" containerName="istio-proxy" Jul 14 18:48:23.738056 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.738042 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5dfc7bc-5a1d-4753-9cb9-286c59ddc6eb" containerName="revoke-keys" Jul 14 18:48:23.738056 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.738051 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a20bdd60-1bd8-413b-87aa-29a19a23d6b6" containerName="istio-proxy" Jul 14 18:48:23.742791 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.742769 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:23.745575 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.745550 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-5bb958-openshift-default-dockercfg-s6z7h\"" Jul 14 18:48:23.753127 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.753081 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr"] Jul 14 18:48:23.787710 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.787676 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bee943-b6a5-4230-b473-670d8af04395-workload-socket\") pod \"e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:23.787856 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.787732 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3bee943-b6a5-4230-b473-670d8af04395-istio-data\") pod \"e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:23.787856 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.787772 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3bee943-b6a5-4230-b473-670d8af04395-istio-token\") pod \"e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:23.787856 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.787843 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3bee943-b6a5-4230-b473-670d8af04395-istiod-ca-cert\") pod \"e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:23.787969 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.787875 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3bee943-b6a5-4230-b473-670d8af04395-istio-podinfo\") pod \"e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:23.787969 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.787896 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3bee943-b6a5-4230-b473-670d8af04395-istio-envoy\") pod \"e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:23.787969 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.787962 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bee943-b6a5-4230-b473-670d8af04395-credential-socket\") pod \"e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:23.788067 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.787989 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rktq\" (UniqueName: \"kubernetes.io/projected/e3bee943-b6a5-4230-b473-670d8af04395-kube-api-access-2rktq\") pod \"e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:23.788067 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.788016 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3bee943-b6a5-4230-b473-670d8af04395-workload-certs\") pod \"e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:23.889359 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.889319 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3bee943-b6a5-4230-b473-670d8af04395-istio-data\") pod \"e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:23.889359 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.889370 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3bee943-b6a5-4230-b473-670d8af04395-istio-token\") pod \"e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:23.889627 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.889410 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3bee943-b6a5-4230-b473-670d8af04395-istiod-ca-cert\") pod \"e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:23.889627 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.889439 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3bee943-b6a5-4230-b473-670d8af04395-istio-podinfo\") pod \"e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:23.889627 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.889466 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3bee943-b6a5-4230-b473-670d8af04395-istio-envoy\") pod \"e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:23.889627 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.889508 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bee943-b6a5-4230-b473-670d8af04395-credential-socket\") pod \"e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:23.889627 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.889531 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rktq\" (UniqueName: \"kubernetes.io/projected/e3bee943-b6a5-4230-b473-670d8af04395-kube-api-access-2rktq\") pod \"e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:23.889627 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.889558 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3bee943-b6a5-4230-b473-670d8af04395-workload-certs\") pod \"e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:23.891312 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.889635 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bee943-b6a5-4230-b473-670d8af04395-workload-socket\") pod \"e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:23.891312 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.889785 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3bee943-b6a5-4230-b473-670d8af04395-istio-data\") pod \"e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:23.891312 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.889945 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bee943-b6a5-4230-b473-670d8af04395-workload-socket\") pod \"e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:23.891312 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.890021 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bee943-b6a5-4230-b473-670d8af04395-credential-socket\") pod \"e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:23.891312 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.890251 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3bee943-b6a5-4230-b473-670d8af04395-workload-certs\") pod \"e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:23.891312 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.890695 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3bee943-b6a5-4230-b473-670d8af04395-istiod-ca-cert\") pod \"e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:23.892308 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.892285 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3bee943-b6a5-4230-b473-670d8af04395-istio-envoy\") pod \"e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:23.892666 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.892647 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3bee943-b6a5-4230-b473-670d8af04395-istio-podinfo\") pod \"e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:23.900545 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.900522 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3bee943-b6a5-4230-b473-670d8af04395-istio-token\") pod \"e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:23.900640 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:23.900586 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rktq\" (UniqueName: \"kubernetes.io/projected/e3bee943-b6a5-4230-b473-670d8af04395-kube-api-access-2rktq\") pod \"e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:24.053923 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:24.053890 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:24.198360 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:24.198335 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr"] Jul 14 18:48:24.200356 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:48:24.200325 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3bee943_b6a5_4230_b473_670d8af04395.slice/crio-5c62d9b49249d116c52339720172d404d1831321c8afd90e72aab6580c08a164 WatchSource:0}: Error finding container 5c62d9b49249d116c52339720172d404d1831321c8afd90e72aab6580c08a164: Status 404 returned error can't find the container with id 5c62d9b49249d116c52339720172d404d1831321c8afd90e72aab6580c08a164 Jul 14 18:48:24.202472 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:24.202441 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:48:24.202539 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:24.202509 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:48:24.202539 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:24.202536 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:48:24.526328 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:24.526294 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" event={"ID":"e3bee943-b6a5-4230-b473-670d8af04395","Type":"ContainerStarted","Data":"3a652e6a2c1d48dedb053236082cbda58e80572e776e3567f34133c13a67509e"} Jul 14 18:48:24.526489 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:24.526345 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" event={"ID":"e3bee943-b6a5-4230-b473-670d8af04395","Type":"ContainerStarted","Data":"5c62d9b49249d116c52339720172d404d1831321c8afd90e72aab6580c08a164"} Jul 14 18:48:24.551279 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:24.551218 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" podStartSLOduration=1.551198383 podStartE2EDuration="1.551198383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:48:24.550324957 +0000 UTC m=+2309.387668213" watchObservedRunningTime="2026-07-14 18:48:24.551198383 +0000 UTC m=+2309.388541641" Jul 14 18:48:25.054870 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:25.054839 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:25.059520 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:25.059498 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:25.531905 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:25.531874 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:25.532913 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:25.532896 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:31.666364 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:31.666330 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:48:39.608083 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:39.608052 2565 generic.go:356] "Generic (PLEG): container finished" podID="e3bee943-b6a5-4230-b473-670d8af04395" containerID="3a652e6a2c1d48dedb053236082cbda58e80572e776e3567f34133c13a67509e" exitCode=0 Jul 14 18:48:39.608494 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:39.608137 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" event={"ID":"e3bee943-b6a5-4230-b473-670d8af04395","Type":"ContainerDied","Data":"3a652e6a2c1d48dedb053236082cbda58e80572e776e3567f34133c13a67509e"} Jul 14 18:48:39.608604 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:39.608587 2565 scope.go:117] "RemoveContainer" containerID="3a652e6a2c1d48dedb053236082cbda58e80572e776e3567f34133c13a67509e" Jul 14 18:48:39.609656 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:39.609633 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:48:39.609713 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:39.609694 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:48:39.609748 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:39.609725 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 18:48:40.532435 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:40.532398 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:40.619172 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:40.619138 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" event={"ID":"e3bee943-b6a5-4230-b473-670d8af04395","Type":"ContainerStarted","Data":"5a0dcf62154395654f016ff912f14e5083802a5c97cc7dcbed0d20abe7f35bbc"} Jul 14 18:48:41.054986 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:41.054944 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:41.056196 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:41.056174 2565 patch_prober.go:28] interesting pod/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 14 18:48:41.056273 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:41.056219 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" podUID="e3bee943-b6a5-4230-b473-670d8af04395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 14 18:48:42.054448 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:42.054414 2565 patch_prober.go:28] interesting pod/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 14 18:48:42.054838 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:42.054466 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" podUID="e3bee943-b6a5-4230-b473-670d8af04395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 14 18:48:43.054616 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:43.054583 2565 patch_prober.go:28] interesting pod/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 14 18:48:43.055022 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:43.054644 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" podUID="e3bee943-b6a5-4230-b473-670d8af04395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 14 18:48:44.055230 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:44.055185 2565 patch_prober.go:28] interesting pod/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 14 18:48:44.055617 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:44.055255 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" podUID="e3bee943-b6a5-4230-b473-670d8af04395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 14 18:48:45.058732 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:45.058701 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:45.059194 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:45.059159 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:45.059899 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:45.059883 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:48:46.563258 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:48:46.563219 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:49:00.656998 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:00.656961 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:49:26.359294 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:26.359263 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr"] Jul 14 18:49:26.361684 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:26.361667 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" Jul 14 18:49:26.364466 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:26.364443 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-41a771\"/\"kube-root-ca.crt\"" Jul 14 18:49:26.365808 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:26.365790 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-41a771\"/\"openshift-service-ca.crt\"" Jul 14 18:49:26.365902 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:26.365790 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-41a771\"/\"auth-test-model-41a771-kserve-self-signed-certs\"" Jul 14 18:49:26.365902 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:26.365790 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-41a771\"/\"default-dockercfg-q6qdm\"" Jul 14 18:49:26.371565 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:26.371540 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr"] Jul 14 18:49:26.482722 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:26.482680 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-tmp-dir\") pod \"auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr\" (UID: \"c1ae7d32-09c8-4fec-bd2f-741f1cb12884\") " pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" Jul 14 18:49:26.482722 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:26.482728 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-home\") pod \"auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr\" (UID: \"c1ae7d32-09c8-4fec-bd2f-741f1cb12884\") " pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" Jul 14 18:49:26.482936 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:26.482747 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-dshm\") pod \"auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr\" (UID: \"c1ae7d32-09c8-4fec-bd2f-741f1cb12884\") " pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" Jul 14 18:49:26.482936 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:26.482828 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgmkl\" (UniqueName: \"kubernetes.io/projected/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-kube-api-access-fgmkl\") pod \"auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr\" (UID: \"c1ae7d32-09c8-4fec-bd2f-741f1cb12884\") " pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" Jul 14 18:49:26.482936 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:26.482855 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-model-cache\") pod \"auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr\" (UID: \"c1ae7d32-09c8-4fec-bd2f-741f1cb12884\") " pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" Jul 14 18:49:26.482936 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:26.482883 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-tls-certs\") pod \"auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr\" (UID: \"c1ae7d32-09c8-4fec-bd2f-741f1cb12884\") " pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" Jul 14 18:49:26.583818 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:26.583782 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-tls-certs\") pod \"auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr\" (UID: \"c1ae7d32-09c8-4fec-bd2f-741f1cb12884\") " pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" Jul 14 18:49:26.584022 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:26.583852 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-tmp-dir\") pod \"auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr\" (UID: \"c1ae7d32-09c8-4fec-bd2f-741f1cb12884\") " pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" Jul 14 18:49:26.584124 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:26.584020 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-home\") pod \"auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr\" (UID: \"c1ae7d32-09c8-4fec-bd2f-741f1cb12884\") " pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" Jul 14 18:49:26.584124 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:26.584057 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-dshm\") pod \"auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr\" (UID: \"c1ae7d32-09c8-4fec-bd2f-741f1cb12884\") " pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" Jul 14 18:49:26.584245 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:26.584143 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgmkl\" (UniqueName: \"kubernetes.io/projected/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-kube-api-access-fgmkl\") pod \"auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr\" (UID: \"c1ae7d32-09c8-4fec-bd2f-741f1cb12884\") " pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" Jul 14 18:49:26.584245 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:26.584172 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-model-cache\") pod \"auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr\" (UID: \"c1ae7d32-09c8-4fec-bd2f-741f1cb12884\") " pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" Jul 14 18:49:26.584245 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:26.584215 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-tmp-dir\") pod \"auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr\" (UID: \"c1ae7d32-09c8-4fec-bd2f-741f1cb12884\") " pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" Jul 14 18:49:26.584412 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:26.584390 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-home\") pod \"auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr\" (UID: \"c1ae7d32-09c8-4fec-bd2f-741f1cb12884\") " pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" Jul 14 18:49:26.584498 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:26.584480 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-model-cache\") pod \"auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr\" (UID: \"c1ae7d32-09c8-4fec-bd2f-741f1cb12884\") " pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" Jul 14 18:49:26.586387 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:26.586353 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-dshm\") pod \"auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr\" (UID: \"c1ae7d32-09c8-4fec-bd2f-741f1cb12884\") " pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" Jul 14 18:49:26.586615 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:26.586599 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-tls-certs\") pod \"auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr\" (UID: \"c1ae7d32-09c8-4fec-bd2f-741f1cb12884\") " pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" Jul 14 18:49:26.597405 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:26.597381 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgmkl\" (UniqueName: \"kubernetes.io/projected/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-kube-api-access-fgmkl\") pod \"auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr\" (UID: \"c1ae7d32-09c8-4fec-bd2f-741f1cb12884\") " pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" Jul 14 18:49:26.671358 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:26.671284 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" Jul 14 18:49:26.805632 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:26.805606 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr"] Jul 14 18:49:26.807518 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:49:26.807493 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1ae7d32_09c8_4fec_bd2f_741f1cb12884.slice/crio-023e9d741cb721e3442a73f9a5328b2b966e6ce2647554d2ecb6c4d09ff3aa79 WatchSource:0}: Error finding container 023e9d741cb721e3442a73f9a5328b2b966e6ce2647554d2ecb6c4d09ff3aa79: Status 404 returned error can't find the container with id 023e9d741cb721e3442a73f9a5328b2b966e6ce2647554d2ecb6c4d09ff3aa79 Jul 14 18:49:26.857799 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:26.857769 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" event={"ID":"c1ae7d32-09c8-4fec-bd2f-741f1cb12884","Type":"ContainerStarted","Data":"023e9d741cb721e3442a73f9a5328b2b966e6ce2647554d2ecb6c4d09ff3aa79"} Jul 14 18:49:27.863989 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:27.863954 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" event={"ID":"c1ae7d32-09c8-4fec-bd2f-741f1cb12884","Type":"ContainerStarted","Data":"6e43d65b9d0dffa4ef6f8349d34786b5fa8539322ae65e3d29a987b2a5d7d903"} Jul 14 18:49:27.884766 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:27.884722 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" podStartSLOduration=1.8847074639999999 podStartE2EDuration="1.884707464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:49:27.883069267 +0000 UTC m=+2372.720412524" watchObservedRunningTime="2026-07-14 18:49:27.884707464 +0000 UTC m=+2372.722050748" Jul 14 18:49:29.952823 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:29.952785 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:49:36.671801 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:36.671766 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" Jul 14 18:49:36.672275 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:36.671816 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" Jul 14 18:49:36.683968 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:36.683946 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" Jul 14 18:49:36.923608 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:36.923520 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" Jul 14 18:49:41.552873 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:41.552832 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:49:55.807425 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:55.807391 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wqpt2_4525dcda-ead6-4800-8d17-58812c289010/console-operator/1.log" Jul 14 18:49:55.809849 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:55.809823 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wqpt2_4525dcda-ead6-4800-8d17-58812c289010/console-operator/1.log" Jul 14 18:49:55.810622 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:55.810599 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f99bz_5f0ecb09-5c7f-4709-9b4a-1a6a03580822/ovn-acl-logging/0.log" Jul 14 18:49:55.813378 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:55.813359 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f99bz_5f0ecb09-5c7f-4709-9b4a-1a6a03580822/ovn-acl-logging/0.log" Jul 14 18:49:57.754843 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:49:57.754810 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:50:15.449156 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:50:15.449122 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:50:32.451055 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:50:32.450971 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:50:50.355908 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:50:50.355870 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:51:08.156011 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:08.155979 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:51:23.296080 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:23.291627 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr"] Jul 14 18:51:26.162475 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:26.162442 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:51:29.861271 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:29.861229 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd"] Jul 14 18:51:29.863841 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:29.863818 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" Jul 14 18:51:29.867886 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:29.867867 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5bb958\"/\"default-dockercfg-xfs96\"" Jul 14 18:51:29.867886 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:29.867875 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-5bb958\"/\"kube-root-ca.crt\"" Jul 14 18:51:29.868124 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:29.867867 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-5bb958\"/\"openshift-service-ca.crt\"" Jul 14 18:51:29.868124 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:29.867886 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5bb958\"/\"sub-test-model-5bb958-kserve-self-signed-certs\"" Jul 14 18:51:29.885457 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:29.885435 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd"] Jul 14 18:51:29.926337 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:29.926301 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2259ad5a-491f-4a95-b01f-fd3a1167dcab-home\") pod \"sub-test-model-5bb958-kserve-85747d6c9c-r7hrd\" (UID: \"2259ad5a-491f-4a95-b01f-fd3a1167dcab\") " pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" Jul 14 18:51:29.926508 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:29.926401 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2259ad5a-491f-4a95-b01f-fd3a1167dcab-dshm\") pod \"sub-test-model-5bb958-kserve-85747d6c9c-r7hrd\" (UID: \"2259ad5a-491f-4a95-b01f-fd3a1167dcab\") " pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" Jul 14 18:51:29.926508 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:29.926458 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2259ad5a-491f-4a95-b01f-fd3a1167dcab-tmp-dir\") pod \"sub-test-model-5bb958-kserve-85747d6c9c-r7hrd\" (UID: \"2259ad5a-491f-4a95-b01f-fd3a1167dcab\") " pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" Jul 14 18:51:29.926508 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:29.926504 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2259ad5a-491f-4a95-b01f-fd3a1167dcab-tls-certs\") pod \"sub-test-model-5bb958-kserve-85747d6c9c-r7hrd\" (UID: \"2259ad5a-491f-4a95-b01f-fd3a1167dcab\") " pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" Jul 14 18:51:29.926645 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:29.926523 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4j2v\" (UniqueName: \"kubernetes.io/projected/2259ad5a-491f-4a95-b01f-fd3a1167dcab-kube-api-access-n4j2v\") pod \"sub-test-model-5bb958-kserve-85747d6c9c-r7hrd\" (UID: \"2259ad5a-491f-4a95-b01f-fd3a1167dcab\") " pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" Jul 14 18:51:29.926645 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:29.926545 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2259ad5a-491f-4a95-b01f-fd3a1167dcab-model-cache\") pod \"sub-test-model-5bb958-kserve-85747d6c9c-r7hrd\" (UID: \"2259ad5a-491f-4a95-b01f-fd3a1167dcab\") " pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" Jul 14 18:51:30.027214 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:30.027178 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2259ad5a-491f-4a95-b01f-fd3a1167dcab-dshm\") pod \"sub-test-model-5bb958-kserve-85747d6c9c-r7hrd\" (UID: \"2259ad5a-491f-4a95-b01f-fd3a1167dcab\") " pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" Jul 14 18:51:30.027386 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:30.027233 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2259ad5a-491f-4a95-b01f-fd3a1167dcab-tmp-dir\") pod \"sub-test-model-5bb958-kserve-85747d6c9c-r7hrd\" (UID: \"2259ad5a-491f-4a95-b01f-fd3a1167dcab\") " pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" Jul 14 18:51:30.027386 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:30.027266 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2259ad5a-491f-4a95-b01f-fd3a1167dcab-tls-certs\") pod \"sub-test-model-5bb958-kserve-85747d6c9c-r7hrd\" (UID: \"2259ad5a-491f-4a95-b01f-fd3a1167dcab\") " pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" Jul 14 18:51:30.027386 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:30.027284 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4j2v\" (UniqueName: \"kubernetes.io/projected/2259ad5a-491f-4a95-b01f-fd3a1167dcab-kube-api-access-n4j2v\") pod \"sub-test-model-5bb958-kserve-85747d6c9c-r7hrd\" (UID: \"2259ad5a-491f-4a95-b01f-fd3a1167dcab\") " pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" Jul 14 18:51:30.027386 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:30.027305 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2259ad5a-491f-4a95-b01f-fd3a1167dcab-model-cache\") pod \"sub-test-model-5bb958-kserve-85747d6c9c-r7hrd\" (UID: \"2259ad5a-491f-4a95-b01f-fd3a1167dcab\") " pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" Jul 14 18:51:30.027606 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:30.027473 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2259ad5a-491f-4a95-b01f-fd3a1167dcab-home\") pod \"sub-test-model-5bb958-kserve-85747d6c9c-r7hrd\" (UID: \"2259ad5a-491f-4a95-b01f-fd3a1167dcab\") " pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" Jul 14 18:51:30.027706 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:30.027681 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2259ad5a-491f-4a95-b01f-fd3a1167dcab-model-cache\") pod \"sub-test-model-5bb958-kserve-85747d6c9c-r7hrd\" (UID: \"2259ad5a-491f-4a95-b01f-fd3a1167dcab\") " pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" Jul 14 18:51:30.027802 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:30.027784 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2259ad5a-491f-4a95-b01f-fd3a1167dcab-tmp-dir\") pod \"sub-test-model-5bb958-kserve-85747d6c9c-r7hrd\" (UID: \"2259ad5a-491f-4a95-b01f-fd3a1167dcab\") " pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" Jul 14 18:51:30.027845 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:30.027811 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2259ad5a-491f-4a95-b01f-fd3a1167dcab-home\") pod \"sub-test-model-5bb958-kserve-85747d6c9c-r7hrd\" (UID: \"2259ad5a-491f-4a95-b01f-fd3a1167dcab\") " pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" Jul 14 18:51:30.029619 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:30.029598 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2259ad5a-491f-4a95-b01f-fd3a1167dcab-dshm\") pod \"sub-test-model-5bb958-kserve-85747d6c9c-r7hrd\" (UID: \"2259ad5a-491f-4a95-b01f-fd3a1167dcab\") " pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" Jul 14 18:51:30.029859 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:30.029839 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2259ad5a-491f-4a95-b01f-fd3a1167dcab-tls-certs\") pod \"sub-test-model-5bb958-kserve-85747d6c9c-r7hrd\" (UID: \"2259ad5a-491f-4a95-b01f-fd3a1167dcab\") " pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" Jul 14 18:51:30.040857 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:30.040833 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4j2v\" (UniqueName: \"kubernetes.io/projected/2259ad5a-491f-4a95-b01f-fd3a1167dcab-kube-api-access-n4j2v\") pod \"sub-test-model-5bb958-kserve-85747d6c9c-r7hrd\" (UID: \"2259ad5a-491f-4a95-b01f-fd3a1167dcab\") " pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" Jul 14 18:51:30.173213 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:30.173127 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" Jul 14 18:51:30.320041 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:30.320016 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd"] Jul 14 18:51:30.321855 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:51:30.321827 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2259ad5a_491f_4a95_b01f_fd3a1167dcab.slice/crio-d8be24b5a60196a315c43dd9c0adbdb30b62c66cf22a724df54683f8d5496247 WatchSource:0}: Error finding container d8be24b5a60196a315c43dd9c0adbdb30b62c66cf22a724df54683f8d5496247: Status 404 returned error can't find the container with id d8be24b5a60196a315c43dd9c0adbdb30b62c66cf22a724df54683f8d5496247 Jul 14 18:51:30.506190 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:30.506074 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" event={"ID":"2259ad5a-491f-4a95-b01f-fd3a1167dcab","Type":"ContainerStarted","Data":"9026f9d1a8b7b1bcd6e4e11e06b6d448c44580a326cbc2198db8271d519217a8"} Jul 14 18:51:30.506190 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:30.506153 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" event={"ID":"2259ad5a-491f-4a95-b01f-fd3a1167dcab","Type":"ContainerStarted","Data":"d8be24b5a60196a315c43dd9c0adbdb30b62c66cf22a724df54683f8d5496247"} Jul 14 18:51:30.533182 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:30.533139 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" podStartSLOduration=1.5331242760000001 podStartE2EDuration="1.533124276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:51:30.531224762 +0000 UTC m=+2495.368568019" watchObservedRunningTime="2026-07-14 18:51:30.533124276 +0000 UTC m=+2495.370467530" Jul 14 18:51:38.312693 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:38.312631 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" podUID="c1ae7d32-09c8-4fec-bd2f-741f1cb12884" containerName="main" containerID="cri-o://6e43d65b9d0dffa4ef6f8349d34786b5fa8539322ae65e3d29a987b2a5d7d903" gracePeriod=45 Jul 14 18:51:38.551846 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:38.551819 2565 generic.go:356] "Generic (PLEG): container finished" podID="c1ae7d32-09c8-4fec-bd2f-741f1cb12884" containerID="6e43d65b9d0dffa4ef6f8349d34786b5fa8539322ae65e3d29a987b2a5d7d903" exitCode=0 Jul 14 18:51:38.551986 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:38.551899 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" event={"ID":"c1ae7d32-09c8-4fec-bd2f-741f1cb12884","Type":"ContainerDied","Data":"6e43d65b9d0dffa4ef6f8349d34786b5fa8539322ae65e3d29a987b2a5d7d903"} Jul 14 18:51:38.551986 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:38.551946 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" event={"ID":"c1ae7d32-09c8-4fec-bd2f-741f1cb12884","Type":"ContainerDied","Data":"023e9d741cb721e3442a73f9a5328b2b966e6ce2647554d2ecb6c4d09ff3aa79"} Jul 14 18:51:38.551986 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:38.551962 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="023e9d741cb721e3442a73f9a5328b2b966e6ce2647554d2ecb6c4d09ff3aa79" Jul 14 18:51:38.561048 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:38.561031 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" Jul 14 18:51:38.600192 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:38.600123 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-tmp-dir\") pod \"c1ae7d32-09c8-4fec-bd2f-741f1cb12884\" (UID: \"c1ae7d32-09c8-4fec-bd2f-741f1cb12884\") " Jul 14 18:51:38.600192 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:38.600184 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgmkl\" (UniqueName: \"kubernetes.io/projected/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-kube-api-access-fgmkl\") pod \"c1ae7d32-09c8-4fec-bd2f-741f1cb12884\" (UID: \"c1ae7d32-09c8-4fec-bd2f-741f1cb12884\") " Jul 14 18:51:38.600382 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:38.600233 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-home\") pod \"c1ae7d32-09c8-4fec-bd2f-741f1cb12884\" (UID: \"c1ae7d32-09c8-4fec-bd2f-741f1cb12884\") " Jul 14 18:51:38.600382 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:38.600256 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-dshm\") pod \"c1ae7d32-09c8-4fec-bd2f-741f1cb12884\" (UID: \"c1ae7d32-09c8-4fec-bd2f-741f1cb12884\") " Jul 14 18:51:38.600382 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:38.600292 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-model-cache\") pod \"c1ae7d32-09c8-4fec-bd2f-741f1cb12884\" (UID: \"c1ae7d32-09c8-4fec-bd2f-741f1cb12884\") " Jul 14 18:51:38.600382 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:38.600345 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-tls-certs\") pod \"c1ae7d32-09c8-4fec-bd2f-741f1cb12884\" (UID: \"c1ae7d32-09c8-4fec-bd2f-741f1cb12884\") " Jul 14 18:51:38.600582 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:38.600420 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c1ae7d32-09c8-4fec-bd2f-741f1cb12884" (UID: "c1ae7d32-09c8-4fec-bd2f-741f1cb12884"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:51:38.600582 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:38.600537 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-home" (OuterVolumeSpecName: "home") pod "c1ae7d32-09c8-4fec-bd2f-741f1cb12884" (UID: "c1ae7d32-09c8-4fec-bd2f-741f1cb12884"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:51:38.600699 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:38.600647 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-model-cache" (OuterVolumeSpecName: "model-cache") pod "c1ae7d32-09c8-4fec-bd2f-741f1cb12884" (UID: "c1ae7d32-09c8-4fec-bd2f-741f1cb12884"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:51:38.600699 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:38.600674 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-tmp-dir\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:51:38.600699 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:38.600692 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-home\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:51:38.602586 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:38.602561 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-dshm" (OuterVolumeSpecName: "dshm") pod "c1ae7d32-09c8-4fec-bd2f-741f1cb12884" (UID: "c1ae7d32-09c8-4fec-bd2f-741f1cb12884"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:51:38.602702 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:38.602636 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-kube-api-access-fgmkl" (OuterVolumeSpecName: "kube-api-access-fgmkl") pod "c1ae7d32-09c8-4fec-bd2f-741f1cb12884" (UID: "c1ae7d32-09c8-4fec-bd2f-741f1cb12884"). InnerVolumeSpecName "kube-api-access-fgmkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:51:38.602855 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:38.602841 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c1ae7d32-09c8-4fec-bd2f-741f1cb12884" (UID: "c1ae7d32-09c8-4fec-bd2f-741f1cb12884"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:51:38.701377 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:38.701350 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-dshm\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:51:38.701377 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:38.701371 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-model-cache\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:51:38.701377 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:38.701381 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-tls-certs\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:51:38.701571 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:38.701390 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgmkl\" (UniqueName: \"kubernetes.io/projected/c1ae7d32-09c8-4fec-bd2f-741f1cb12884-kube-api-access-fgmkl\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:51:39.556926 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:39.556893 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr" Jul 14 18:51:39.588024 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:39.587983 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr"] Jul 14 18:51:39.595577 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:39.595549 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-41a771/auth-test-model-41a771-kserve-5cd4cfcfdc-g88qr"] Jul 14 18:51:39.689702 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:39.689675 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1ae7d32-09c8-4fec-bd2f-741f1cb12884" path="/var/lib/kubelet/pods/c1ae7d32-09c8-4fec-bd2f-741f1cb12884/volumes" Jul 14 18:51:40.173292 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:40.173259 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" Jul 14 18:51:40.173292 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:40.173299 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" Jul 14 18:51:40.185356 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:40.185331 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" Jul 14 18:51:40.573077 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:40.573050 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" Jul 14 18:51:41.768384 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:41.768348 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:51:55.919444 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:55.919419 2565 scope.go:117] "RemoveContainer" containerID="dd4a058f2f33d9f7c2db5d6d2dbefe040ac366a666bfab68989ae4a3930e7db2" Jul 14 18:51:55.927540 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:51:55.927522 2565 scope.go:117] "RemoveContainer" containerID="13e196df360a8463f0c0e5b773ba2a45e152d7eebc2db5f1403fedadc309760d" Jul 14 18:52:00.666140 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:00.666110 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:52:30.576233 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:30.576202 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:52:45.255600 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:45.255566 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:52:51.405493 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:51.405453 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd"] Jul 14 18:52:56.549215 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:56.549177 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k"] Jul 14 18:52:56.549756 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:56.549734 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1ae7d32-09c8-4fec-bd2f-741f1cb12884" containerName="main" Jul 14 18:52:56.549845 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:56.549759 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1ae7d32-09c8-4fec-bd2f-741f1cb12884" containerName="main" Jul 14 18:52:56.549897 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:56.549861 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1ae7d32-09c8-4fec-bd2f-741f1cb12884" containerName="main" Jul 14 18:52:56.552087 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:56.552065 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" Jul 14 18:52:56.554642 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:56.554622 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5bb958\"/\"rate-test-model-5bb958-kserve-self-signed-certs\"" Jul 14 18:52:56.562478 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:56.562457 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k"] Jul 14 18:52:56.667443 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:56.667412 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6399ded-66c1-40cf-8c21-3448b573249a-home\") pod \"rate-test-model-5bb958-kserve-5b44695579-4bv7k\" (UID: \"c6399ded-66c1-40cf-8c21-3448b573249a\") " pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" Jul 14 18:52:56.667443 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:56.667449 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6399ded-66c1-40cf-8c21-3448b573249a-dshm\") pod \"rate-test-model-5bb958-kserve-5b44695579-4bv7k\" (UID: \"c6399ded-66c1-40cf-8c21-3448b573249a\") " pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" Jul 14 18:52:56.667708 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:56.667544 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6399ded-66c1-40cf-8c21-3448b573249a-tmp-dir\") pod \"rate-test-model-5bb958-kserve-5b44695579-4bv7k\" (UID: \"c6399ded-66c1-40cf-8c21-3448b573249a\") " pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" Jul 14 18:52:56.667708 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:56.667617 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6399ded-66c1-40cf-8c21-3448b573249a-model-cache\") pod \"rate-test-model-5bb958-kserve-5b44695579-4bv7k\" (UID: \"c6399ded-66c1-40cf-8c21-3448b573249a\") " pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" Jul 14 18:52:56.667708 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:56.667679 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6399ded-66c1-40cf-8c21-3448b573249a-tls-certs\") pod \"rate-test-model-5bb958-kserve-5b44695579-4bv7k\" (UID: \"c6399ded-66c1-40cf-8c21-3448b573249a\") " pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" Jul 14 18:52:56.667840 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:56.667723 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v26z\" (UniqueName: \"kubernetes.io/projected/c6399ded-66c1-40cf-8c21-3448b573249a-kube-api-access-2v26z\") pod \"rate-test-model-5bb958-kserve-5b44695579-4bv7k\" (UID: \"c6399ded-66c1-40cf-8c21-3448b573249a\") " pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" Jul 14 18:52:56.768953 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:56.768914 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6399ded-66c1-40cf-8c21-3448b573249a-tls-certs\") pod \"rate-test-model-5bb958-kserve-5b44695579-4bv7k\" (UID: \"c6399ded-66c1-40cf-8c21-3448b573249a\") " pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" Jul 14 18:52:56.769721 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:56.769699 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v26z\" (UniqueName: \"kubernetes.io/projected/c6399ded-66c1-40cf-8c21-3448b573249a-kube-api-access-2v26z\") pod \"rate-test-model-5bb958-kserve-5b44695579-4bv7k\" (UID: \"c6399ded-66c1-40cf-8c21-3448b573249a\") " pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" Jul 14 18:52:56.770271 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:56.770251 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6399ded-66c1-40cf-8c21-3448b573249a-home\") pod \"rate-test-model-5bb958-kserve-5b44695579-4bv7k\" (UID: \"c6399ded-66c1-40cf-8c21-3448b573249a\") " pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" Jul 14 18:52:56.770741 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:56.770727 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6399ded-66c1-40cf-8c21-3448b573249a-dshm\") pod \"rate-test-model-5bb958-kserve-5b44695579-4bv7k\" (UID: \"c6399ded-66c1-40cf-8c21-3448b573249a\") " pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" Jul 14 18:52:56.771141 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:56.770687 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6399ded-66c1-40cf-8c21-3448b573249a-home\") pod \"rate-test-model-5bb958-kserve-5b44695579-4bv7k\" (UID: \"c6399ded-66c1-40cf-8c21-3448b573249a\") " pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" Jul 14 18:52:56.771314 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:56.771166 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6399ded-66c1-40cf-8c21-3448b573249a-tmp-dir\") pod \"rate-test-model-5bb958-kserve-5b44695579-4bv7k\" (UID: \"c6399ded-66c1-40cf-8c21-3448b573249a\") " pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" Jul 14 18:52:56.771493 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:56.771380 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6399ded-66c1-40cf-8c21-3448b573249a-tmp-dir\") pod \"rate-test-model-5bb958-kserve-5b44695579-4bv7k\" (UID: \"c6399ded-66c1-40cf-8c21-3448b573249a\") " pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" Jul 14 18:52:56.771604 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:56.771588 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6399ded-66c1-40cf-8c21-3448b573249a-model-cache\") pod \"rate-test-model-5bb958-kserve-5b44695579-4bv7k\" (UID: \"c6399ded-66c1-40cf-8c21-3448b573249a\") " pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" Jul 14 18:52:56.772044 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:56.771912 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6399ded-66c1-40cf-8c21-3448b573249a-model-cache\") pod \"rate-test-model-5bb958-kserve-5b44695579-4bv7k\" (UID: \"c6399ded-66c1-40cf-8c21-3448b573249a\") " pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" Jul 14 18:52:56.773880 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:56.773860 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6399ded-66c1-40cf-8c21-3448b573249a-tls-certs\") pod \"rate-test-model-5bb958-kserve-5b44695579-4bv7k\" (UID: \"c6399ded-66c1-40cf-8c21-3448b573249a\") " pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" Jul 14 18:52:56.774470 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:56.774455 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6399ded-66c1-40cf-8c21-3448b573249a-dshm\") pod \"rate-test-model-5bb958-kserve-5b44695579-4bv7k\" (UID: \"c6399ded-66c1-40cf-8c21-3448b573249a\") " pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" Jul 14 18:52:56.779616 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:56.779596 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v26z\" (UniqueName: \"kubernetes.io/projected/c6399ded-66c1-40cf-8c21-3448b573249a-kube-api-access-2v26z\") pod \"rate-test-model-5bb958-kserve-5b44695579-4bv7k\" (UID: \"c6399ded-66c1-40cf-8c21-3448b573249a\") " pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" Jul 14 18:52:56.861846 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:56.861757 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" Jul 14 18:52:56.997901 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:56.997877 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k"] Jul 14 18:52:57.000054 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:52:57.000031 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6399ded_66c1_40cf_8c21_3448b573249a.slice/crio-18a8b5d1c5fe3266fec929ab50041fba1aef736a44875183515381d2623d6d47 WatchSource:0}: Error finding container 18a8b5d1c5fe3266fec929ab50041fba1aef736a44875183515381d2623d6d47: Status 404 returned error can't find the container with id 18a8b5d1c5fe3266fec929ab50041fba1aef736a44875183515381d2623d6d47 Jul 14 18:52:57.952665 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:57.952633 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" event={"ID":"c6399ded-66c1-40cf-8c21-3448b573249a","Type":"ContainerStarted","Data":"8bc9e77613c562b8e9f0240221a7983a1c694870d8420414b3af00e2fbe7ad30"} Jul 14 18:52:57.952665 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:57.952673 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" event={"ID":"c6399ded-66c1-40cf-8c21-3448b573249a","Type":"ContainerStarted","Data":"18a8b5d1c5fe3266fec929ab50041fba1aef736a44875183515381d2623d6d47"} Jul 14 18:52:57.975757 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:52:57.975707 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" podStartSLOduration=1.975690436 podStartE2EDuration="1.975690436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:52:57.973322028 +0000 UTC m=+2582.810665308" watchObservedRunningTime="2026-07-14 18:52:57.975690436 +0000 UTC m=+2582.813033695" Jul 14 18:53:05.059836 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:05.059800 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:53:06.425256 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:06.425207 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" podUID="2259ad5a-491f-4a95-b01f-fd3a1167dcab" containerName="main" containerID="cri-o://9026f9d1a8b7b1bcd6e4e11e06b6d448c44580a326cbc2198db8271d519217a8" gracePeriod=45 Jul 14 18:53:06.676764 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:06.676697 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" Jul 14 18:53:06.761353 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:06.761325 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2259ad5a-491f-4a95-b01f-fd3a1167dcab-dshm\") pod \"2259ad5a-491f-4a95-b01f-fd3a1167dcab\" (UID: \"2259ad5a-491f-4a95-b01f-fd3a1167dcab\") " Jul 14 18:53:06.761517 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:06.761382 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2259ad5a-491f-4a95-b01f-fd3a1167dcab-tmp-dir\") pod \"2259ad5a-491f-4a95-b01f-fd3a1167dcab\" (UID: \"2259ad5a-491f-4a95-b01f-fd3a1167dcab\") " Jul 14 18:53:06.761517 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:06.761411 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2259ad5a-491f-4a95-b01f-fd3a1167dcab-tls-certs\") pod \"2259ad5a-491f-4a95-b01f-fd3a1167dcab\" (UID: \"2259ad5a-491f-4a95-b01f-fd3a1167dcab\") " Jul 14 18:53:06.761517 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:06.761427 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2259ad5a-491f-4a95-b01f-fd3a1167dcab-model-cache\") pod \"2259ad5a-491f-4a95-b01f-fd3a1167dcab\" (UID: \"2259ad5a-491f-4a95-b01f-fd3a1167dcab\") " Jul 14 18:53:06.761517 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:06.761461 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4j2v\" (UniqueName: \"kubernetes.io/projected/2259ad5a-491f-4a95-b01f-fd3a1167dcab-kube-api-access-n4j2v\") pod \"2259ad5a-491f-4a95-b01f-fd3a1167dcab\" (UID: \"2259ad5a-491f-4a95-b01f-fd3a1167dcab\") " Jul 14 18:53:06.761517 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:06.761498 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2259ad5a-491f-4a95-b01f-fd3a1167dcab-home\") pod \"2259ad5a-491f-4a95-b01f-fd3a1167dcab\" (UID: \"2259ad5a-491f-4a95-b01f-fd3a1167dcab\") " Jul 14 18:53:06.761785 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:06.761724 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2259ad5a-491f-4a95-b01f-fd3a1167dcab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2259ad5a-491f-4a95-b01f-fd3a1167dcab" (UID: "2259ad5a-491f-4a95-b01f-fd3a1167dcab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:53:06.761785 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:06.761740 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2259ad5a-491f-4a95-b01f-fd3a1167dcab-model-cache" (OuterVolumeSpecName: "model-cache") pod "2259ad5a-491f-4a95-b01f-fd3a1167dcab" (UID: "2259ad5a-491f-4a95-b01f-fd3a1167dcab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:53:06.761883 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:06.761826 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2259ad5a-491f-4a95-b01f-fd3a1167dcab-home" (OuterVolumeSpecName: "home") pod "2259ad5a-491f-4a95-b01f-fd3a1167dcab" (UID: "2259ad5a-491f-4a95-b01f-fd3a1167dcab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:53:06.763774 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:06.763753 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2259ad5a-491f-4a95-b01f-fd3a1167dcab-dshm" (OuterVolumeSpecName: "dshm") pod "2259ad5a-491f-4a95-b01f-fd3a1167dcab" (UID: "2259ad5a-491f-4a95-b01f-fd3a1167dcab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:53:06.763888 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:06.763773 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2259ad5a-491f-4a95-b01f-fd3a1167dcab-kube-api-access-n4j2v" (OuterVolumeSpecName: "kube-api-access-n4j2v") pod "2259ad5a-491f-4a95-b01f-fd3a1167dcab" (UID: "2259ad5a-491f-4a95-b01f-fd3a1167dcab"). InnerVolumeSpecName "kube-api-access-n4j2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:53:06.763888 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:06.763780 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2259ad5a-491f-4a95-b01f-fd3a1167dcab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2259ad5a-491f-4a95-b01f-fd3a1167dcab" (UID: "2259ad5a-491f-4a95-b01f-fd3a1167dcab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:53:06.862336 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:06.862294 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" Jul 14 18:53:06.862473 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:06.862363 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" Jul 14 18:53:06.862736 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:06.862719 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2259ad5a-491f-4a95-b01f-fd3a1167dcab-dshm\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:53:06.862866 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:06.862848 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2259ad5a-491f-4a95-b01f-fd3a1167dcab-tmp-dir\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:53:06.862962 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:06.862947 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2259ad5a-491f-4a95-b01f-fd3a1167dcab-tls-certs\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:53:06.862996 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:06.862968 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2259ad5a-491f-4a95-b01f-fd3a1167dcab-model-cache\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:53:06.862996 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:06.862982 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4j2v\" (UniqueName: \"kubernetes.io/projected/2259ad5a-491f-4a95-b01f-fd3a1167dcab-kube-api-access-n4j2v\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:53:06.863057 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:06.862995 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2259ad5a-491f-4a95-b01f-fd3a1167dcab-home\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:53:06.874571 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:06.874551 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" Jul 14 18:53:07.000158 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:07.000059 2565 generic.go:356] "Generic (PLEG): container finished" podID="2259ad5a-491f-4a95-b01f-fd3a1167dcab" containerID="9026f9d1a8b7b1bcd6e4e11e06b6d448c44580a326cbc2198db8271d519217a8" exitCode=0 Jul 14 18:53:07.000158 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:07.000113 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" Jul 14 18:53:07.000158 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:07.000149 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" event={"ID":"2259ad5a-491f-4a95-b01f-fd3a1167dcab","Type":"ContainerDied","Data":"9026f9d1a8b7b1bcd6e4e11e06b6d448c44580a326cbc2198db8271d519217a8"} Jul 14 18:53:07.000414 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:07.000198 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd" event={"ID":"2259ad5a-491f-4a95-b01f-fd3a1167dcab","Type":"ContainerDied","Data":"d8be24b5a60196a315c43dd9c0adbdb30b62c66cf22a724df54683f8d5496247"} Jul 14 18:53:07.000414 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:07.000219 2565 scope.go:117] "RemoveContainer" containerID="9026f9d1a8b7b1bcd6e4e11e06b6d448c44580a326cbc2198db8271d519217a8" Jul 14 18:53:07.009763 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:07.009744 2565 scope.go:117] "RemoveContainer" containerID="9026f9d1a8b7b1bcd6e4e11e06b6d448c44580a326cbc2198db8271d519217a8" Jul 14 18:53:07.010016 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:53:07.009998 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9026f9d1a8b7b1bcd6e4e11e06b6d448c44580a326cbc2198db8271d519217a8\": container with ID starting with 9026f9d1a8b7b1bcd6e4e11e06b6d448c44580a326cbc2198db8271d519217a8 not found: ID does not exist" containerID="9026f9d1a8b7b1bcd6e4e11e06b6d448c44580a326cbc2198db8271d519217a8" Jul 14 18:53:07.010088 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:07.010024 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9026f9d1a8b7b1bcd6e4e11e06b6d448c44580a326cbc2198db8271d519217a8"} err="failed to get container status \"9026f9d1a8b7b1bcd6e4e11e06b6d448c44580a326cbc2198db8271d519217a8\": rpc error: code = NotFound desc = could not find container \"9026f9d1a8b7b1bcd6e4e11e06b6d448c44580a326cbc2198db8271d519217a8\": container with ID starting with 9026f9d1a8b7b1bcd6e4e11e06b6d448c44580a326cbc2198db8271d519217a8 not found: ID does not exist" Jul 14 18:53:07.011979 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:07.011961 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" Jul 14 18:53:07.025050 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:07.025025 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd"] Jul 14 18:53:07.029567 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:07.029546 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-5bb958/sub-test-model-5bb958-kserve-85747d6c9c-r7hrd"] Jul 14 18:53:07.690713 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:07.690681 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2259ad5a-491f-4a95-b01f-fd3a1167dcab" path="/var/lib/kubelet/pods/2259ad5a-491f-4a95-b01f-fd3a1167dcab/volumes" Jul 14 18:53:37.257550 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:37.257517 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:53:51.854801 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:53:51.854766 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:54:07.354705 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:07.354664 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:54:15.278370 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:15.278333 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k"] Jul 14 18:54:21.555394 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:21.555360 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:54:30.301201 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:30.301136 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" podUID="c6399ded-66c1-40cf-8c21-3448b573249a" containerName="main" containerID="cri-o://8bc9e77613c562b8e9f0240221a7983a1c694870d8420414b3af00e2fbe7ad30" gracePeriod=45 Jul 14 18:54:30.444872 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:30.444832 2565 generic.go:356] "Generic (PLEG): container finished" podID="c6399ded-66c1-40cf-8c21-3448b573249a" containerID="8bc9e77613c562b8e9f0240221a7983a1c694870d8420414b3af00e2fbe7ad30" exitCode=0 Jul 14 18:54:30.445009 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:30.444898 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" event={"ID":"c6399ded-66c1-40cf-8c21-3448b573249a","Type":"ContainerDied","Data":"8bc9e77613c562b8e9f0240221a7983a1c694870d8420414b3af00e2fbe7ad30"} Jul 14 18:54:30.555053 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:30.554997 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" Jul 14 18:54:30.586450 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:30.586424 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6399ded-66c1-40cf-8c21-3448b573249a-tls-certs\") pod \"c6399ded-66c1-40cf-8c21-3448b573249a\" (UID: \"c6399ded-66c1-40cf-8c21-3448b573249a\") " Jul 14 18:54:30.586611 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:30.586469 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6399ded-66c1-40cf-8c21-3448b573249a-dshm\") pod \"c6399ded-66c1-40cf-8c21-3448b573249a\" (UID: \"c6399ded-66c1-40cf-8c21-3448b573249a\") " Jul 14 18:54:30.586696 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:30.586616 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6399ded-66c1-40cf-8c21-3448b573249a-tmp-dir\") pod \"c6399ded-66c1-40cf-8c21-3448b573249a\" (UID: \"c6399ded-66c1-40cf-8c21-3448b573249a\") " Jul 14 18:54:30.586696 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:30.586689 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2v26z\" (UniqueName: \"kubernetes.io/projected/c6399ded-66c1-40cf-8c21-3448b573249a-kube-api-access-2v26z\") pod \"c6399ded-66c1-40cf-8c21-3448b573249a\" (UID: \"c6399ded-66c1-40cf-8c21-3448b573249a\") " Jul 14 18:54:30.586809 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:30.586720 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6399ded-66c1-40cf-8c21-3448b573249a-home\") pod \"c6399ded-66c1-40cf-8c21-3448b573249a\" (UID: \"c6399ded-66c1-40cf-8c21-3448b573249a\") " Jul 14 18:54:30.586809 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:30.586768 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6399ded-66c1-40cf-8c21-3448b573249a-model-cache\") pod \"c6399ded-66c1-40cf-8c21-3448b573249a\" (UID: \"c6399ded-66c1-40cf-8c21-3448b573249a\") " Jul 14 18:54:30.587559 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:30.587528 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6399ded-66c1-40cf-8c21-3448b573249a-home" (OuterVolumeSpecName: "home") pod "c6399ded-66c1-40cf-8c21-3448b573249a" (UID: "c6399ded-66c1-40cf-8c21-3448b573249a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:54:30.587714 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:30.587590 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6399ded-66c1-40cf-8c21-3448b573249a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c6399ded-66c1-40cf-8c21-3448b573249a" (UID: "c6399ded-66c1-40cf-8c21-3448b573249a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:54:30.587714 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:30.587630 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6399ded-66c1-40cf-8c21-3448b573249a-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6399ded-66c1-40cf-8c21-3448b573249a" (UID: "c6399ded-66c1-40cf-8c21-3448b573249a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:54:30.588941 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:30.588916 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6399ded-66c1-40cf-8c21-3448b573249a-dshm" (OuterVolumeSpecName: "dshm") pod "c6399ded-66c1-40cf-8c21-3448b573249a" (UID: "c6399ded-66c1-40cf-8c21-3448b573249a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:54:30.589214 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:30.589194 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6399ded-66c1-40cf-8c21-3448b573249a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6399ded-66c1-40cf-8c21-3448b573249a" (UID: "c6399ded-66c1-40cf-8c21-3448b573249a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:54:30.589310 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:30.589202 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6399ded-66c1-40cf-8c21-3448b573249a-kube-api-access-2v26z" (OuterVolumeSpecName: "kube-api-access-2v26z") pod "c6399ded-66c1-40cf-8c21-3448b573249a" (UID: "c6399ded-66c1-40cf-8c21-3448b573249a"). InnerVolumeSpecName "kube-api-access-2v26z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:54:30.687581 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:30.687547 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6399ded-66c1-40cf-8c21-3448b573249a-tls-certs\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:54:30.687581 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:30.687581 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6399ded-66c1-40cf-8c21-3448b573249a-dshm\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:54:30.687581 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:30.687593 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6399ded-66c1-40cf-8c21-3448b573249a-tmp-dir\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:54:30.687864 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:30.687605 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v26z\" (UniqueName: \"kubernetes.io/projected/c6399ded-66c1-40cf-8c21-3448b573249a-kube-api-access-2v26z\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:54:30.687864 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:30.687617 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6399ded-66c1-40cf-8c21-3448b573249a-home\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:54:30.687864 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:30.687629 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6399ded-66c1-40cf-8c21-3448b573249a-model-cache\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:54:31.451899 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:31.451874 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" Jul 14 18:54:31.452394 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:31.451875 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k" event={"ID":"c6399ded-66c1-40cf-8c21-3448b573249a","Type":"ContainerDied","Data":"18a8b5d1c5fe3266fec929ab50041fba1aef736a44875183515381d2623d6d47"} Jul 14 18:54:31.452394 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:31.452010 2565 scope.go:117] "RemoveContainer" containerID="8bc9e77613c562b8e9f0240221a7983a1c694870d8420414b3af00e2fbe7ad30" Jul 14 18:54:31.477245 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:31.477215 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k"] Jul 14 18:54:31.482237 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:31.482211 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-5bb958/rate-test-model-5bb958-kserve-5b44695579-4bv7k"] Jul 14 18:54:31.689949 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:31.689916 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6399ded-66c1-40cf-8c21-3448b573249a" path="/var/lib/kubelet/pods/c6399ded-66c1-40cf-8c21-3448b573249a/volumes" Jul 14 18:54:35.560964 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:35.560927 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5bb958-k5x9f"] Jul 14 18:54:35.561364 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:35.561311 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6399ded-66c1-40cf-8c21-3448b573249a" containerName="main" Jul 14 18:54:35.561364 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:35.561324 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6399ded-66c1-40cf-8c21-3448b573249a" containerName="main" Jul 14 18:54:35.561364 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:35.561339 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2259ad5a-491f-4a95-b01f-fd3a1167dcab" containerName="main" Jul 14 18:54:35.561364 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:35.561344 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2259ad5a-491f-4a95-b01f-fd3a1167dcab" containerName="main" Jul 14 18:54:35.561497 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:35.561396 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6399ded-66c1-40cf-8c21-3448b573249a" containerName="main" Jul 14 18:54:35.561497 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:35.561404 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2259ad5a-491f-4a95-b01f-fd3a1167dcab" containerName="main" Jul 14 18:54:35.564330 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:35.564314 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5bb958-k5x9f" Jul 14 18:54:35.567235 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:35.567211 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 18:54:35.567355 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:35.567249 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9jplk\"" Jul 14 18:54:35.576232 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:35.576208 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5bb958-k5x9f"] Jul 14 18:54:35.629929 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:35.629892 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c6feb8c-6640-4a2f-aa19-6474eaf2a826-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-5bb958-k5x9f\" (UID: \"2c6feb8c-6640-4a2f-aa19-6474eaf2a826\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5bb958-k5x9f" Jul 14 18:54:35.731183 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:35.731151 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c6feb8c-6640-4a2f-aa19-6474eaf2a826-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-5bb958-k5x9f\" (UID: \"2c6feb8c-6640-4a2f-aa19-6474eaf2a826\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5bb958-k5x9f" Jul 14 18:54:35.731804 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:35.731786 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c6feb8c-6640-4a2f-aa19-6474eaf2a826-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-5bb958-k5x9f\" (UID: \"2c6feb8c-6640-4a2f-aa19-6474eaf2a826\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5bb958-k5x9f" Jul 14 18:54:35.873933 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:35.873851 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5bb958-k5x9f" Jul 14 18:54:36.008437 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:36.008406 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5bb958-k5x9f"] Jul 14 18:54:36.010544 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:54:36.010516 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c6feb8c_6640_4a2f_aa19_6474eaf2a826.slice/crio-489249ba6ba9cdc67c26b5a67a6bce8b47d747382d75a2722e4d65ee2325a5e1 WatchSource:0}: Error finding container 489249ba6ba9cdc67c26b5a67a6bce8b47d747382d75a2722e4d65ee2325a5e1: Status 404 returned error can't find the container with id 489249ba6ba9cdc67c26b5a67a6bce8b47d747382d75a2722e4d65ee2325a5e1 Jul 14 18:54:36.478820 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:36.478730 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5bb958-k5x9f" event={"ID":"2c6feb8c-6640-4a2f-aa19-6474eaf2a826","Type":"ContainerStarted","Data":"d67c15125b8c76cfb4c69765df30d326966be447abf3ec84e5fa01c9d6ca4d4e"} Jul 14 18:54:36.478820 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:36.478783 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5bb958-k5x9f" event={"ID":"2c6feb8c-6640-4a2f-aa19-6474eaf2a826","Type":"ContainerStarted","Data":"489249ba6ba9cdc67c26b5a67a6bce8b47d747382d75a2722e4d65ee2325a5e1"} Jul 14 18:54:36.497066 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:36.497011 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5bb958-k5x9f" podStartSLOduration=1.496995127 podStartE2EDuration="1.496995127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:54:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:54:36.496529877 +0000 UTC m=+2681.333873136" watchObservedRunningTime="2026-07-14 18:54:36.496995127 +0000 UTC m=+2681.334338384" Jul 14 18:54:37.485465 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:37.485436 2565 generic.go:356] "Generic (PLEG): container finished" podID="2c6feb8c-6640-4a2f-aa19-6474eaf2a826" containerID="d67c15125b8c76cfb4c69765df30d326966be447abf3ec84e5fa01c9d6ca4d4e" exitCode=0 Jul 14 18:54:37.485866 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:37.485523 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5bb958-k5x9f" event={"ID":"2c6feb8c-6640-4a2f-aa19-6474eaf2a826","Type":"ContainerDied","Data":"d67c15125b8c76cfb4c69765df30d326966be447abf3ec84e5fa01c9d6ca4d4e"} Jul 14 18:54:38.620787 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:38.620762 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5bb958-k5x9f" Jul 14 18:54:38.656750 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:38.656726 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c6feb8c-6640-4a2f-aa19-6474eaf2a826-maas-api-service-ca\") pod \"2c6feb8c-6640-4a2f-aa19-6474eaf2a826\" (UID: \"2c6feb8c-6640-4a2f-aa19-6474eaf2a826\") " Jul 14 18:54:38.657134 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:38.657084 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c6feb8c-6640-4a2f-aa19-6474eaf2a826-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2c6feb8c-6640-4a2f-aa19-6474eaf2a826" (UID: "2c6feb8c-6640-4a2f-aa19-6474eaf2a826"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:54:38.758378 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:38.758292 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c6feb8c-6640-4a2f-aa19-6474eaf2a826-maas-api-service-ca\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:54:39.497900 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:39.497869 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5bb958-k5x9f" Jul 14 18:54:39.498072 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:39.497900 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5bb958-k5x9f" event={"ID":"2c6feb8c-6640-4a2f-aa19-6474eaf2a826","Type":"ContainerDied","Data":"489249ba6ba9cdc67c26b5a67a6bce8b47d747382d75a2722e4d65ee2325a5e1"} Jul 14 18:54:39.498072 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:39.497967 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="489249ba6ba9cdc67c26b5a67a6bce8b47d747382d75a2722e4d65ee2325a5e1" Jul 14 18:54:49.650066 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:49.650030 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:54:55.853306 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:55.853280 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wqpt2_4525dcda-ead6-4800-8d17-58812c289010/console-operator/1.log" Jul 14 18:54:55.856337 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:55.856314 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f99bz_5f0ecb09-5c7f-4709-9b4a-1a6a03580822/ovn-acl-logging/0.log" Jul 14 18:54:55.857117 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:55.857081 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wqpt2_4525dcda-ead6-4800-8d17-58812c289010/console-operator/1.log" Jul 14 18:54:55.860333 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:54:55.860317 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f99bz_5f0ecb09-5c7f-4709-9b4a-1a6a03580822/ovn-acl-logging/0.log" Jul 14 18:55:01.527009 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:01.526975 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr"] Jul 14 18:55:01.527504 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:01.527274 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" podUID="e3bee943-b6a5-4230-b473-670d8af04395" containerName="istio-proxy" containerID="cri-o://5a0dcf62154395654f016ff912f14e5083802a5c97cc7dcbed0d20abe7f35bbc" gracePeriod=30 Jul 14 18:55:01.972174 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:01.972076 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-41a771-zg9j5"] Jul 14 18:55:01.972491 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:01.972477 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c6feb8c-6640-4a2f-aa19-6474eaf2a826" containerName="revoke-keys" Jul 14 18:55:01.972546 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:01.972494 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c6feb8c-6640-4a2f-aa19-6474eaf2a826" containerName="revoke-keys" Jul 14 18:55:01.972594 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:01.972571 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c6feb8c-6640-4a2f-aa19-6474eaf2a826" containerName="revoke-keys" Jul 14 18:55:01.977050 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:01.977030 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-41a771-zg9j5" Jul 14 18:55:01.979717 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:01.979694 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 18:55:01.979806 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:01.979694 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9jplk\"" Jul 14 18:55:01.985849 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:01.985825 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-41a771-zg9j5"] Jul 14 18:55:02.053944 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:02.053913 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e26c721b-7977-4dd3-b2c6-a41a85d29e65-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-41a771-zg9j5\" (UID: \"e26c721b-7977-4dd3-b2c6-a41a85d29e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-41a771-zg9j5" Jul 14 18:55:02.155014 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:02.154978 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e26c721b-7977-4dd3-b2c6-a41a85d29e65-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-41a771-zg9j5\" (UID: \"e26c721b-7977-4dd3-b2c6-a41a85d29e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-41a771-zg9j5" Jul 14 18:55:02.155650 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:02.155630 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e26c721b-7977-4dd3-b2c6-a41a85d29e65-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-41a771-zg9j5\" (UID: \"e26c721b-7977-4dd3-b2c6-a41a85d29e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-41a771-zg9j5" Jul 14 18:55:02.288658 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:02.288624 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-41a771-zg9j5" Jul 14 18:55:02.428593 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:02.428409 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-41a771-zg9j5"] Jul 14 18:55:02.430863 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:55:02.430837 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode26c721b_7977_4dd3_b2c6_a41a85d29e65.slice/crio-b409b4dbe5fdc3990f7fde9558d299e3eed39fd99f9b928152046842b1fcc670 WatchSource:0}: Error finding container b409b4dbe5fdc3990f7fde9558d299e3eed39fd99f9b928152046842b1fcc670: Status 404 returned error can't find the container with id b409b4dbe5fdc3990f7fde9558d299e3eed39fd99f9b928152046842b1fcc670 Jul 14 18:55:02.617310 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:02.617272 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-41a771-zg9j5" event={"ID":"e26c721b-7977-4dd3-b2c6-a41a85d29e65","Type":"ContainerStarted","Data":"b409b4dbe5fdc3990f7fde9558d299e3eed39fd99f9b928152046842b1fcc670"} Jul 14 18:55:03.623450 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:03.623413 2565 generic.go:356] "Generic (PLEG): container finished" podID="e26c721b-7977-4dd3-b2c6-a41a85d29e65" containerID="0196477db972fdaa429a9c867bf5b7780066bbb25b9196190dfa601165d6718e" exitCode=0 Jul 14 18:55:03.623839 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:03.623498 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-41a771-zg9j5" event={"ID":"e26c721b-7977-4dd3-b2c6-a41a85d29e65","Type":"ContainerDied","Data":"0196477db972fdaa429a9c867bf5b7780066bbb25b9196190dfa601165d6718e"} Jul 14 18:55:04.762246 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:04.762223 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-41a771-zg9j5" Jul 14 18:55:04.875563 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:04.875529 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e26c721b-7977-4dd3-b2c6-a41a85d29e65-maas-api-service-ca\") pod \"e26c721b-7977-4dd3-b2c6-a41a85d29e65\" (UID: \"e26c721b-7977-4dd3-b2c6-a41a85d29e65\") " Jul 14 18:55:04.875884 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:04.875860 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e26c721b-7977-4dd3-b2c6-a41a85d29e65-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e26c721b-7977-4dd3-b2c6-a41a85d29e65" (UID: "e26c721b-7977-4dd3-b2c6-a41a85d29e65"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:55:04.976320 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:04.976238 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e26c721b-7977-4dd3-b2c6-a41a85d29e65-maas-api-service-ca\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:55:05.633675 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:05.633643 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-41a771-zg9j5" event={"ID":"e26c721b-7977-4dd3-b2c6-a41a85d29e65","Type":"ContainerDied","Data":"b409b4dbe5fdc3990f7fde9558d299e3eed39fd99f9b928152046842b1fcc670"} Jul 14 18:55:05.633675 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:05.633673 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-41a771-zg9j5" Jul 14 18:55:05.633898 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:05.633689 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b409b4dbe5fdc3990f7fde9558d299e3eed39fd99f9b928152046842b1fcc670" Jul 14 18:55:06.792340 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:06.792280 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:55:06.890924 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:06.890895 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3bee943-b6a5-4230-b473-670d8af04395-workload-certs\") pod \"e3bee943-b6a5-4230-b473-670d8af04395\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " Jul 14 18:55:06.891118 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:06.890937 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3bee943-b6a5-4230-b473-670d8af04395-istio-token\") pod \"e3bee943-b6a5-4230-b473-670d8af04395\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " Jul 14 18:55:06.891118 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:06.890957 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3bee943-b6a5-4230-b473-670d8af04395-istiod-ca-cert\") pod \"e3bee943-b6a5-4230-b473-670d8af04395\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " Jul 14 18:55:06.891118 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:06.890975 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bee943-b6a5-4230-b473-670d8af04395-credential-socket\") pod \"e3bee943-b6a5-4230-b473-670d8af04395\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " Jul 14 18:55:06.891118 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:06.891006 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rktq\" (UniqueName: \"kubernetes.io/projected/e3bee943-b6a5-4230-b473-670d8af04395-kube-api-access-2rktq\") pod \"e3bee943-b6a5-4230-b473-670d8af04395\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " Jul 14 18:55:06.891118 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:06.891063 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bee943-b6a5-4230-b473-670d8af04395-workload-socket\") pod \"e3bee943-b6a5-4230-b473-670d8af04395\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " Jul 14 18:55:06.891420 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:06.891127 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3bee943-b6a5-4230-b473-670d8af04395-istio-podinfo\") pod \"e3bee943-b6a5-4230-b473-670d8af04395\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " Jul 14 18:55:06.891420 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:06.891154 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3bee943-b6a5-4230-b473-670d8af04395-istio-data\") pod \"e3bee943-b6a5-4230-b473-670d8af04395\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " Jul 14 18:55:06.891420 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:06.891212 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3bee943-b6a5-4230-b473-670d8af04395-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e3bee943-b6a5-4230-b473-670d8af04395" (UID: "e3bee943-b6a5-4230-b473-670d8af04395"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:55:06.891420 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:06.891226 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3bee943-b6a5-4230-b473-670d8af04395-istio-envoy\") pod \"e3bee943-b6a5-4230-b473-670d8af04395\" (UID: \"e3bee943-b6a5-4230-b473-670d8af04395\") " Jul 14 18:55:06.891420 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:06.891365 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3bee943-b6a5-4230-b473-670d8af04395-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e3bee943-b6a5-4230-b473-670d8af04395" (UID: "e3bee943-b6a5-4230-b473-670d8af04395"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:55:06.891420 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:06.891394 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3bee943-b6a5-4230-b473-670d8af04395-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e3bee943-b6a5-4230-b473-670d8af04395" (UID: "e3bee943-b6a5-4230-b473-670d8af04395"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:55:06.891720 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:06.891524 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3bee943-b6a5-4230-b473-670d8af04395-workload-certs\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:55:06.891720 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:06.891545 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bee943-b6a5-4230-b473-670d8af04395-credential-socket\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:55:06.891720 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:06.891559 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bee943-b6a5-4230-b473-670d8af04395-workload-socket\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:55:06.891720 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:06.891670 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3bee943-b6a5-4230-b473-670d8af04395-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e3bee943-b6a5-4230-b473-670d8af04395" (UID: "e3bee943-b6a5-4230-b473-670d8af04395"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:55:06.892333 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:06.892307 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3bee943-b6a5-4230-b473-670d8af04395-istio-data" (OuterVolumeSpecName: "istio-data") pod "e3bee943-b6a5-4230-b473-670d8af04395" (UID: "e3bee943-b6a5-4230-b473-670d8af04395"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:55:06.893365 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:06.893335 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3bee943-b6a5-4230-b473-670d8af04395-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e3bee943-b6a5-4230-b473-670d8af04395" (UID: "e3bee943-b6a5-4230-b473-670d8af04395"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 18:55:06.893466 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:06.893414 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3bee943-b6a5-4230-b473-670d8af04395-istio-token" (OuterVolumeSpecName: "istio-token") pod "e3bee943-b6a5-4230-b473-670d8af04395" (UID: "e3bee943-b6a5-4230-b473-670d8af04395"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:55:06.893466 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:06.893412 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3bee943-b6a5-4230-b473-670d8af04395-kube-api-access-2rktq" (OuterVolumeSpecName: "kube-api-access-2rktq") pod "e3bee943-b6a5-4230-b473-670d8af04395" (UID: "e3bee943-b6a5-4230-b473-670d8af04395"). InnerVolumeSpecName "kube-api-access-2rktq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:55:06.893543 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:06.893505 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3bee943-b6a5-4230-b473-670d8af04395-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e3bee943-b6a5-4230-b473-670d8af04395" (UID: "e3bee943-b6a5-4230-b473-670d8af04395"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:55:06.992896 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:06.992868 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3bee943-b6a5-4230-b473-670d8af04395-istio-envoy\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:55:06.992896 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:06.992894 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3bee943-b6a5-4230-b473-670d8af04395-istio-token\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:55:06.992896 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:06.992902 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3bee943-b6a5-4230-b473-670d8af04395-istiod-ca-cert\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:55:06.993084 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:06.992910 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rktq\" (UniqueName: \"kubernetes.io/projected/e3bee943-b6a5-4230-b473-670d8af04395-kube-api-access-2rktq\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:55:06.993084 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:06.992919 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3bee943-b6a5-4230-b473-670d8af04395-istio-podinfo\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:55:06.993084 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:06.992926 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3bee943-b6a5-4230-b473-670d8af04395-istio-data\") on node \"ip-10-0-136-25.ec2.internal\" DevicePath \"\"" Jul 14 18:55:07.644789 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:07.644760 2565 generic.go:356] "Generic (PLEG): container finished" podID="e3bee943-b6a5-4230-b473-670d8af04395" containerID="5a0dcf62154395654f016ff912f14e5083802a5c97cc7dcbed0d20abe7f35bbc" exitCode=0 Jul 14 18:55:07.645011 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:07.644807 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" Jul 14 18:55:07.645011 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:07.644833 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" event={"ID":"e3bee943-b6a5-4230-b473-670d8af04395","Type":"ContainerDied","Data":"5a0dcf62154395654f016ff912f14e5083802a5c97cc7dcbed0d20abe7f35bbc"} Jul 14 18:55:07.645011 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:07.644898 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr" event={"ID":"e3bee943-b6a5-4230-b473-670d8af04395","Type":"ContainerDied","Data":"5c62d9b49249d116c52339720172d404d1831321c8afd90e72aab6580c08a164"} Jul 14 18:55:07.645011 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:07.644919 2565 scope.go:117] "RemoveContainer" containerID="5a0dcf62154395654f016ff912f14e5083802a5c97cc7dcbed0d20abe7f35bbc" Jul 14 18:55:07.654594 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:07.654577 2565 scope.go:117] "RemoveContainer" containerID="3a652e6a2c1d48dedb053236082cbda58e80572e776e3567f34133c13a67509e" Jul 14 18:55:07.662568 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:07.662550 2565 scope.go:117] "RemoveContainer" containerID="5a0dcf62154395654f016ff912f14e5083802a5c97cc7dcbed0d20abe7f35bbc" Jul 14 18:55:07.662825 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:55:07.662804 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a0dcf62154395654f016ff912f14e5083802a5c97cc7dcbed0d20abe7f35bbc\": container with ID starting with 5a0dcf62154395654f016ff912f14e5083802a5c97cc7dcbed0d20abe7f35bbc not found: ID does not exist" containerID="5a0dcf62154395654f016ff912f14e5083802a5c97cc7dcbed0d20abe7f35bbc" Jul 14 18:55:07.662917 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:07.662830 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a0dcf62154395654f016ff912f14e5083802a5c97cc7dcbed0d20abe7f35bbc"} err="failed to get container status \"5a0dcf62154395654f016ff912f14e5083802a5c97cc7dcbed0d20abe7f35bbc\": rpc error: code = NotFound desc = could not find container \"5a0dcf62154395654f016ff912f14e5083802a5c97cc7dcbed0d20abe7f35bbc\": container with ID starting with 5a0dcf62154395654f016ff912f14e5083802a5c97cc7dcbed0d20abe7f35bbc not found: ID does not exist" Jul 14 18:55:07.662917 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:07.662848 2565 scope.go:117] "RemoveContainer" containerID="3a652e6a2c1d48dedb053236082cbda58e80572e776e3567f34133c13a67509e" Jul 14 18:55:07.663077 ip-10-0-136-25 kubenswrapper[2565]: E0714 18:55:07.663056 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a652e6a2c1d48dedb053236082cbda58e80572e776e3567f34133c13a67509e\": container with ID starting with 3a652e6a2c1d48dedb053236082cbda58e80572e776e3567f34133c13a67509e not found: ID does not exist" containerID="3a652e6a2c1d48dedb053236082cbda58e80572e776e3567f34133c13a67509e" Jul 14 18:55:07.663140 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:07.663084 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a652e6a2c1d48dedb053236082cbda58e80572e776e3567f34133c13a67509e"} err="failed to get container status \"3a652e6a2c1d48dedb053236082cbda58e80572e776e3567f34133c13a67509e\": rpc error: code = NotFound desc = could not find container \"3a652e6a2c1d48dedb053236082cbda58e80572e776e3567f34133c13a67509e\": container with ID starting with 3a652e6a2c1d48dedb053236082cbda58e80572e776e3567f34133c13a67509e not found: ID does not exist" Jul 14 18:55:07.670227 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:07.670206 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr"] Jul 14 18:55:07.679874 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:07.679852 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-5bb958-openshift-default-65c994c7bc-fcxgr"] Jul 14 18:55:07.689448 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:07.689423 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3bee943-b6a5-4230-b473-670d8af04395" path="/var/lib/kubelet/pods/e3bee943-b6a5-4230-b473-670d8af04395/volumes" Jul 14 18:55:15.760255 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:15.760219 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ztrvp"] Jul 14 18:55:34.054206 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:34.054173 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-jhz62_3c94b2dd-bbdc-4b9b-a696-658cc441ed9e/manager/0.log" Jul 14 18:55:34.184647 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:34.184617 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-vx6wz_ced6629e-359f-404f-9598-86ee1c9351a9/manager/0.log" Jul 14 18:55:34.606393 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:34.606363 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-wq6tl_8816c36a-81c8-4f59-8f8c-8d813e09d253/manager/0.log" Jul 14 18:55:34.734423 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:34.734390 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-557b7f646c-84xtc_a2cc16bb-ccb9-4852-aaa2-00a337a95ee2/copy-manifests/0.log" Jul 14 18:55:34.749444 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:34.749410 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-557b7f646c-84xtc_a2cc16bb-ccb9-4852-aaa2-00a337a95ee2/manager/0.log" Jul 14 18:55:35.662235 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:35.662208 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29734245-26cjg_f03021d3-89fb-4bc0-91cd-b7fa20c16e3c/cleanup/0.log" Jul 14 18:55:35.788577 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:35.788544 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-1808df45-4tx94_3b965b04-799c-4da6-96e2-8c5de94a548e/revoke-keys/1.log" Jul 14 18:55:35.917435 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:35.917353 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-f37aedf1-8fzrl_acde68d5-161c-46d6-a261-a8c73dec7739/revoke-keys/0.log" Jul 14 18:55:36.058980 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:36.058951 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-fade18-bbvlm_1f52e563-9f2b-4a80-bced-88c919bc3e7b/revoke-keys/0.log" Jul 14 18:55:36.190162 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:36.190055 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-fde6bc-4dw7l_f5dfc7bc-5a1d-4753-9cb9-286c59ddc6eb/revoke-keys/0.log" Jul 14 18:55:36.329612 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:36.329586 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-e2e80b9d-d8jfl_be5b1dcb-aff2-4483-b651-6d77460e946e/revoke-keys/0.log" Jul 14 18:55:36.453487 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:36.453403 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-1f114d-2qsqw_4c6ca669-5fc0-403c-8627-10ad92528262/revoke-keys/0.log" Jul 14 18:55:36.573849 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:36.573820 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-7965f3-kn29s_53c7b8ef-ff7a-4a99-870b-e0e5104e7258/revoke-keys/0.log" Jul 14 18:55:36.695522 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:36.695495 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-499cb51a-gk9lz_6e8e2ff3-29b7-41ba-99ba-aafaedc1e795/revoke-keys/0.log" Jul 14 18:55:36.817842 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:36.817811 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-467eca84-cvmcl_9c6396bc-0d5a-4819-9d4d-fbdb43cf740d/revoke-keys/0.log" Jul 14 18:55:36.944152 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:36.944122 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-c99ca1c1-z9h7b_21a9f60c-0c98-4374-a7e1-95f35a4a6c66/revoke-keys/0.log" Jul 14 18:55:37.068753 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:37.068680 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-5bb958-k5x9f_2c6feb8c-6640-4a2f-aa19-6474eaf2a826/revoke-keys/0.log" Jul 14 18:55:37.195039 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:37.195014 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-41a771-zg9j5_e26c721b-7977-4dd3-b2c6-a41a85d29e65/revoke-keys/0.log" Jul 14 18:55:39.384274 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:39.384231 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-bzrwb_cebd3785-87d9-41f2-883c-f96424b89823/manager/0.log" Jul 14 18:55:39.508877 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:39.508851 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-ztrvp_bf5effef-70b1-40d2-9959-537f4ca0f0a1/limitador/0.log" Jul 14 18:55:40.164294 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:40.164262 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-npbch_45ec5309-1cfc-4477-aa0f-082ce7485a9c/discovery/0.log" Jul 14 18:55:40.414619 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:40.414535 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-565f78b694-g6wdp_80e4139d-1310-414d-89c6-b775fad20eff/kube-auth-proxy/0.log" Jul 14 18:55:40.542921 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:40.542891 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-sv6fr_6ac2ef1e-824e-482f-8087-dc90fa894ebd/istio-proxy/1.log" Jul 14 18:55:40.935640 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:40.935608 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5977bb4ff6-wbzmj_7b2746f9-dc43-4311-aa5b-ad3bc351deb3/router/0.log" Jul 14 18:55:41.555836 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:41.555808 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-lpdjq_76b21e27-b560-4e7e-a2c8-7455b0017a30/main/0.log" Jul 14 18:55:49.975888 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:49.975854 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6czgz_015d11c8-2009-4c76-815a-c2d87c42e7b5/global-pull-secret-syncer/0.log" Jul 14 18:55:50.227895 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:50.227818 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sgdnj_e3d8f9e9-f69e-4a3d-b688-ff64a5ec495e/konnectivity-agent/0.log" Jul 14 18:55:50.305724 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:50.305696 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-25.ec2.internal_03b994a435c6ac42ecdd2d02de3d23bb/haproxy/0.log" Jul 14 18:55:55.887692 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:55.887656 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-bzrwb_cebd3785-87d9-41f2-883c-f96424b89823/manager/0.log" Jul 14 18:55:55.927232 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:55.927210 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-ztrvp_bf5effef-70b1-40d2-9959-537f4ca0f0a1/limitador/0.log" Jul 14 18:55:55.984732 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:55.982793 2565 scope.go:117] "RemoveContainer" containerID="6e43d65b9d0dffa4ef6f8349d34786b5fa8539322ae65e3d29a987b2a5d7d903" Jul 14 18:55:57.872445 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:57.872416 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-7jp42_1c9bdfa6-8e91-4ba7-a3fc-cdeb38bac452/cluster-monitoring-operator/0.log" Jul 14 18:55:58.149130 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:58.149034 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-phnhn_09b26111-d3a6-47f0-98eb-73a1f0008b00/node-exporter/0.log" Jul 14 18:55:58.177721 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:58.177693 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-phnhn_09b26111-d3a6-47f0-98eb-73a1f0008b00/kube-rbac-proxy/0.log" Jul 14 18:55:58.212291 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:58.212265 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-phnhn_09b26111-d3a6-47f0-98eb-73a1f0008b00/init-textfile/0.log" Jul 14 18:55:59.708847 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:59.706146 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-492wj/perf-node-gather-daemonset-98xnr"] Jul 14 18:55:59.708847 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:59.706934 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3bee943-b6a5-4230-b473-670d8af04395" containerName="istio-proxy" Jul 14 18:55:59.708847 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:59.706951 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3bee943-b6a5-4230-b473-670d8af04395" containerName="istio-proxy" Jul 14 18:55:59.708847 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:59.706983 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3bee943-b6a5-4230-b473-670d8af04395" containerName="istio-proxy" Jul 14 18:55:59.708847 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:59.706991 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3bee943-b6a5-4230-b473-670d8af04395" containerName="istio-proxy" Jul 14 18:55:59.708847 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:59.707022 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e26c721b-7977-4dd3-b2c6-a41a85d29e65" containerName="revoke-keys" Jul 14 18:55:59.708847 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:59.707031 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e26c721b-7977-4dd3-b2c6-a41a85d29e65" containerName="revoke-keys" Jul 14 18:55:59.708847 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:59.707198 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3bee943-b6a5-4230-b473-670d8af04395" containerName="istio-proxy" Jul 14 18:55:59.708847 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:59.707221 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e26c721b-7977-4dd3-b2c6-a41a85d29e65" containerName="revoke-keys" Jul 14 18:55:59.714330 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:59.714291 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-492wj/perf-node-gather-daemonset-98xnr" Jul 14 18:55:59.717493 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:59.717473 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-492wj\"/\"default-dockercfg-n6svq\"" Jul 14 18:55:59.717637 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:59.717519 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-492wj\"/\"kube-root-ca.crt\"" Jul 14 18:55:59.718722 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:59.718699 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-492wj\"/\"openshift-service-ca.crt\"" Jul 14 18:55:59.721481 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:59.721462 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-492wj/perf-node-gather-daemonset-98xnr"] Jul 14 18:55:59.830856 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:59.830823 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htqbn\" (UniqueName: \"kubernetes.io/projected/4edb7320-7c26-4131-8d09-3109282391a9-kube-api-access-htqbn\") pod \"perf-node-gather-daemonset-98xnr\" (UID: \"4edb7320-7c26-4131-8d09-3109282391a9\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-98xnr" Jul 14 18:55:59.831035 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:59.830936 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4edb7320-7c26-4131-8d09-3109282391a9-podres\") pod \"perf-node-gather-daemonset-98xnr\" (UID: \"4edb7320-7c26-4131-8d09-3109282391a9\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-98xnr" Jul 14 18:55:59.831035 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:59.831007 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4edb7320-7c26-4131-8d09-3109282391a9-sys\") pod \"perf-node-gather-daemonset-98xnr\" (UID: \"4edb7320-7c26-4131-8d09-3109282391a9\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-98xnr" Jul 14 18:55:59.831191 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:59.831043 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4edb7320-7c26-4131-8d09-3109282391a9-lib-modules\") pod \"perf-node-gather-daemonset-98xnr\" (UID: \"4edb7320-7c26-4131-8d09-3109282391a9\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-98xnr" Jul 14 18:55:59.831191 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:59.831093 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4edb7320-7c26-4131-8d09-3109282391a9-proc\") pod \"perf-node-gather-daemonset-98xnr\" (UID: \"4edb7320-7c26-4131-8d09-3109282391a9\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-98xnr" Jul 14 18:55:59.931961 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:59.931933 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4edb7320-7c26-4131-8d09-3109282391a9-proc\") pod \"perf-node-gather-daemonset-98xnr\" (UID: \"4edb7320-7c26-4131-8d09-3109282391a9\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-98xnr" Jul 14 18:55:59.932122 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:59.932007 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htqbn\" (UniqueName: \"kubernetes.io/projected/4edb7320-7c26-4131-8d09-3109282391a9-kube-api-access-htqbn\") pod \"perf-node-gather-daemonset-98xnr\" (UID: \"4edb7320-7c26-4131-8d09-3109282391a9\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-98xnr" Jul 14 18:55:59.932122 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:59.932051 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4edb7320-7c26-4131-8d09-3109282391a9-podres\") pod \"perf-node-gather-daemonset-98xnr\" (UID: \"4edb7320-7c26-4131-8d09-3109282391a9\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-98xnr" Jul 14 18:55:59.932122 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:59.932053 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4edb7320-7c26-4131-8d09-3109282391a9-proc\") pod \"perf-node-gather-daemonset-98xnr\" (UID: \"4edb7320-7c26-4131-8d09-3109282391a9\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-98xnr" Jul 14 18:55:59.932238 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:59.932126 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4edb7320-7c26-4131-8d09-3109282391a9-sys\") pod \"perf-node-gather-daemonset-98xnr\" (UID: \"4edb7320-7c26-4131-8d09-3109282391a9\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-98xnr" Jul 14 18:55:59.932238 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:59.932157 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4edb7320-7c26-4131-8d09-3109282391a9-lib-modules\") pod \"perf-node-gather-daemonset-98xnr\" (UID: \"4edb7320-7c26-4131-8d09-3109282391a9\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-98xnr" Jul 14 18:55:59.932238 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:59.932196 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4edb7320-7c26-4131-8d09-3109282391a9-podres\") pod \"perf-node-gather-daemonset-98xnr\" (UID: \"4edb7320-7c26-4131-8d09-3109282391a9\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-98xnr" Jul 14 18:55:59.932238 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:59.932225 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4edb7320-7c26-4131-8d09-3109282391a9-sys\") pod \"perf-node-gather-daemonset-98xnr\" (UID: \"4edb7320-7c26-4131-8d09-3109282391a9\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-98xnr" Jul 14 18:55:59.932376 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:59.932253 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4edb7320-7c26-4131-8d09-3109282391a9-lib-modules\") pod \"perf-node-gather-daemonset-98xnr\" (UID: \"4edb7320-7c26-4131-8d09-3109282391a9\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-98xnr" Jul 14 18:55:59.940873 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:55:59.940841 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htqbn\" (UniqueName: \"kubernetes.io/projected/4edb7320-7c26-4131-8d09-3109282391a9-kube-api-access-htqbn\") pod \"perf-node-gather-daemonset-98xnr\" (UID: \"4edb7320-7c26-4131-8d09-3109282391a9\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-98xnr" Jul 14 18:56:00.025019 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:00.024995 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-492wj/perf-node-gather-daemonset-98xnr" Jul 14 18:56:00.142384 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:00.142355 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-lgbnj_3dd54969-694e-4b8c-8a8c-21412a2c954e/networking-console-plugin/0.log" Jul 14 18:56:00.159472 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:00.159450 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-492wj/perf-node-gather-daemonset-98xnr"] Jul 14 18:56:00.161247 ip-10-0-136-25 kubenswrapper[2565]: W0714 18:56:00.161222 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4edb7320_7c26_4131_8d09_3109282391a9.slice/crio-33a6084e57a54f70bbeae9e6c3da48c0c40cb6c1d8ac2a764110faa100b53805 WatchSource:0}: Error finding container 33a6084e57a54f70bbeae9e6c3da48c0c40cb6c1d8ac2a764110faa100b53805: Status 404 returned error can't find the container with id 33a6084e57a54f70bbeae9e6c3da48c0c40cb6c1d8ac2a764110faa100b53805 Jul 14 18:56:00.808660 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:00.808634 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wqpt2_4525dcda-ead6-4800-8d17-58812c289010/console-operator/1.log" Jul 14 18:56:00.812605 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:00.812580 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-wqpt2_4525dcda-ead6-4800-8d17-58812c289010/console-operator/2.log" Jul 14 18:56:00.911938 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:00.911909 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-492wj/perf-node-gather-daemonset-98xnr" event={"ID":"4edb7320-7c26-4131-8d09-3109282391a9","Type":"ContainerStarted","Data":"4d98e2879ba1e7bf5459950a9113cbc0bfd4cb809427390045350d4e52c72ed0"} Jul 14 18:56:00.912122 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:00.911958 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-492wj/perf-node-gather-daemonset-98xnr" event={"ID":"4edb7320-7c26-4131-8d09-3109282391a9","Type":"ContainerStarted","Data":"33a6084e57a54f70bbeae9e6c3da48c0c40cb6c1d8ac2a764110faa100b53805"} Jul 14 18:56:00.912122 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:00.911994 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-492wj/perf-node-gather-daemonset-98xnr" Jul 14 18:56:00.932667 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:00.932614 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-492wj/perf-node-gather-daemonset-98xnr" podStartSLOduration=1.93259855 podStartE2EDuration="1.93259855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:56:00.93188633 +0000 UTC m=+2765.769229609" watchObservedRunningTime="2026-07-14 18:56:00.93259855 +0000 UTC m=+2765.769941806" Jul 14 18:56:01.983720 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:01.983683 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-qldqp_938c2572-16f6-4f69-90e7-f33f7e46d4b4/volume-data-source-validator/0.log" Jul 14 18:56:03.106422 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:03.106391 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6k9qk_44267c25-6aee-4e48-93da-3b222a618eff/dns/0.log" Jul 14 18:56:03.129770 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:03.129745 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6k9qk_44267c25-6aee-4e48-93da-3b222a618eff/kube-rbac-proxy/0.log" Jul 14 18:56:03.306133 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:03.306088 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ds4bd_0089b7f3-1d21-4a3d-b5d8-fc1cd289f06e/dns-node-resolver/0.log" Jul 14 18:56:03.964610 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:03.964579 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qx9gz_87ca2cc2-58d4-4899-8bc1-2b5bd93ebe6f/node-ca/0.log" Jul 14 18:56:05.182050 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:05.182018 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-npbch_45ec5309-1cfc-4477-aa0f-082ce7485a9c/discovery/0.log" Jul 14 18:56:05.232811 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:05.232784 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-565f78b694-g6wdp_80e4139d-1310-414d-89c6-b775fad20eff/kube-auth-proxy/0.log" Jul 14 18:56:05.265554 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:05.265525 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-sv6fr_6ac2ef1e-824e-482f-8087-dc90fa894ebd/istio-proxy/0.log" Jul 14 18:56:05.294112 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:05.294073 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-sv6fr_6ac2ef1e-824e-482f-8087-dc90fa894ebd/istio-proxy/1.log" Jul 14 18:56:05.387651 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:05.387627 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5977bb4ff6-wbzmj_7b2746f9-dc43-4311-aa5b-ad3bc351deb3/router/0.log" Jul 14 18:56:06.155043 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:06.155020 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jmt26_211b531e-f105-44a4-934f-d2e97e981b84/serve-healthcheck-canary/0.log" Jul 14 18:56:06.710476 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:06.710445 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-mqqk9_29cfec31-7adb-41a3-965d-11c1d199b9c3/insights-operator/0.log" Jul 14 18:56:06.710989 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:06.710975 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-mqqk9_29cfec31-7adb-41a3-965d-11c1d199b9c3/insights-operator/1.log" Jul 14 18:56:06.734962 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:06.734943 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9g9f7_9c5e5426-5ad9-4ecc-93aa-577db1551181/kube-rbac-proxy/0.log" Jul 14 18:56:06.761021 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:06.761001 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9g9f7_9c5e5426-5ad9-4ecc-93aa-577db1551181/exporter/0.log" Jul 14 18:56:06.787627 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:06.787609 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9g9f7_9c5e5426-5ad9-4ecc-93aa-577db1551181/extractor/0.log" Jul 14 18:56:06.927062 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:06.927038 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-492wj/perf-node-gather-daemonset-98xnr" Jul 14 18:56:09.403492 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:09.403391 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-jhz62_3c94b2dd-bbdc-4b9b-a696-658cc441ed9e/manager/0.log" Jul 14 18:56:09.538606 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:09.538579 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-vx6wz_ced6629e-359f-404f-9598-86ee1c9351a9/manager/0.log" Jul 14 18:56:09.746634 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:09.746559 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-wq6tl_8816c36a-81c8-4f59-8f8c-8d813e09d253/manager/0.log" Jul 14 18:56:09.824383 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:09.824354 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-557b7f646c-84xtc_a2cc16bb-ccb9-4852-aaa2-00a337a95ee2/manager/0.log" Jul 14 18:56:09.844788 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:09.844764 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-557b7f646c-84xtc_a2cc16bb-ccb9-4852-aaa2-00a337a95ee2/copy-manifests/0.log" Jul 14 18:56:10.426520 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:10.426485 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-npbch_45ec5309-1cfc-4477-aa0f-082ce7485a9c/discovery/0.log" Jul 14 18:56:10.457142 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:10.457119 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-565f78b694-g6wdp_80e4139d-1310-414d-89c6-b775fad20eff/kube-auth-proxy/0.log" Jul 14 18:56:10.479866 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:10.479838 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-sv6fr_6ac2ef1e-824e-482f-8087-dc90fa894ebd/istio-proxy/0.log" Jul 14 18:56:10.506796 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:10.506765 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-sv6fr_6ac2ef1e-824e-482f-8087-dc90fa894ebd/istio-proxy/1.log" Jul 14 18:56:10.563842 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:10.563821 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5977bb4ff6-wbzmj_7b2746f9-dc43-4311-aa5b-ad3bc351deb3/router/0.log" Jul 14 18:56:11.909309 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:11.909278 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-npbch_45ec5309-1cfc-4477-aa0f-082ce7485a9c/discovery/0.log" Jul 14 18:56:11.940304 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:11.940264 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-565f78b694-g6wdp_80e4139d-1310-414d-89c6-b775fad20eff/kube-auth-proxy/0.log" Jul 14 18:56:11.962964 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:11.962940 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-sv6fr_6ac2ef1e-824e-482f-8087-dc90fa894ebd/istio-proxy/0.log" Jul 14 18:56:11.993596 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:11.993575 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-sv6fr_6ac2ef1e-824e-482f-8087-dc90fa894ebd/istio-proxy/1.log" Jul 14 18:56:12.056757 ip-10-0-136-25 kubenswrapper[2565]: I0714 18:56:12.056734 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5977bb4ff6-wbzmj_7b2746f9-dc43-4311-aa5b-ad3bc351deb3/router/0.log"