Jul 18 02:37:56.360607 ip-10-0-140-4 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 18 02:37:56.360619 ip-10-0-140-4 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 18 02:37:56.360626 ip-10-0-140-4 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 18 02:37:56.360832 ip-10-0-140-4 systemd[1]: Failed to start Kubernetes Kubelet. Jul 18 02:38:06.561975 ip-10-0-140-4 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 18 02:38:06.561990 ip-10-0-140-4 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot f87717faf57149088785e41d3cbd5025 -- Jul 18 02:39:36.270442 ip-10-0-140-4 systemd[1]: Starting Kubernetes Kubelet... Jul 18 02:39:36.715059 ip-10-0-140-4 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 18 02:39:36.715059 ip-10-0-140-4 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 18 02:39:36.715059 ip-10-0-140-4 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 18 02:39:36.715059 ip-10-0-140-4 kubenswrapper[2585]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 18 02:39:36.715545 ip-10-0-140-4 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 18 02:39:36.716206 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.716123 2585 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 18 02:39:36.721043 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721029 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 02:39:36.721043 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721045 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 02:39:36.721143 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721050 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 02:39:36.721143 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721054 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 02:39:36.721143 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721058 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 02:39:36.721143 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721062 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 02:39:36.721143 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721066 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 02:39:36.721143 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721070 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 02:39:36.721143 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721077 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 02:39:36.721143 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721083 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 02:39:36.721143 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721088 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 02:39:36.721143 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721093 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 02:39:36.721143 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721097 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 02:39:36.721143 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721101 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 02:39:36.721143 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721105 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 02:39:36.721143 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721109 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 02:39:36.721143 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721113 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 02:39:36.721143 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721117 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 02:39:36.721143 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721123 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 02:39:36.721143 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721128 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 02:39:36.721143 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721132 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 02:39:36.721902 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721148 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 02:39:36.721902 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721152 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 02:39:36.721902 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721157 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 02:39:36.721902 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721161 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 02:39:36.721902 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721165 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 02:39:36.721902 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721169 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 02:39:36.721902 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721173 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 02:39:36.721902 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721178 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 02:39:36.721902 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721198 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 02:39:36.721902 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721202 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 02:39:36.721902 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721206 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 02:39:36.721902 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721220 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 02:39:36.721902 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721226 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 02:39:36.721902 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721230 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 02:39:36.721902 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721234 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 02:39:36.721902 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721239 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 02:39:36.721902 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721243 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 02:39:36.721902 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721246 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 02:39:36.721902 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721250 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 02:39:36.722826 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721254 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 02:39:36.722826 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721259 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 02:39:36.722826 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721263 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 02:39:36.722826 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721267 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 02:39:36.722826 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721271 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 02:39:36.722826 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721275 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 02:39:36.722826 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721280 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 02:39:36.722826 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721284 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 02:39:36.722826 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721288 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 02:39:36.722826 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721292 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 02:39:36.722826 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721296 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 02:39:36.722826 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721300 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 02:39:36.722826 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721304 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 02:39:36.722826 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721308 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 02:39:36.722826 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721313 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 02:39:36.722826 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721317 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 02:39:36.722826 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721322 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 02:39:36.722826 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721326 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 02:39:36.722826 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721330 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 02:39:36.722826 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721334 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 02:39:36.723632 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721338 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 02:39:36.723632 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721343 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 02:39:36.723632 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721348 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 02:39:36.723632 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721352 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 02:39:36.723632 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721356 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 02:39:36.723632 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721361 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 02:39:36.723632 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721365 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 02:39:36.723632 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721369 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 02:39:36.723632 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721373 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 02:39:36.723632 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721377 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 02:39:36.723632 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721384 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 02:39:36.723632 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721389 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 02:39:36.723632 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721393 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 02:39:36.723632 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721397 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 02:39:36.723632 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721402 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 18 02:39:36.723632 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721406 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 02:39:36.723632 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721410 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 02:39:36.723632 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721414 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 02:39:36.723632 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721418 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 02:39:36.724429 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721425 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 02:39:36.724429 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721430 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 02:39:36.724429 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721435 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 02:39:36.724429 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721440 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 02:39:36.724429 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721444 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 02:39:36.724429 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721449 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 02:39:36.724429 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721453 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 02:39:36.724429 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721457 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 02:39:36.724429 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721462 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 02:39:36.724429 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721467 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 02:39:36.724429 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721471 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 02:39:36.724429 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721475 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 02:39:36.724429 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721479 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 02:39:36.724429 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721483 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 02:39:36.724429 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721488 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 02:39:36.724429 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721492 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 02:39:36.724429 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721495 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 02:39:36.724429 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721500 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 02:39:36.724429 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721504 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 02:39:36.724429 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.721509 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 02:39:36.725249 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722644 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 02:39:36.725249 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722654 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 02:39:36.725249 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722659 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 02:39:36.725249 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722664 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 02:39:36.725249 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722669 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 02:39:36.725249 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722682 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 02:39:36.725249 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722688 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 02:39:36.725249 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722696 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 02:39:36.725249 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722700 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 02:39:36.725249 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722704 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 02:39:36.725249 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722708 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 02:39:36.725249 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722712 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 02:39:36.725249 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722716 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 02:39:36.725249 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722720 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 02:39:36.725249 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722724 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 02:39:36.725249 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722728 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 02:39:36.725249 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722732 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 02:39:36.725249 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722736 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 02:39:36.725249 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722740 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 02:39:36.725249 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722744 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 02:39:36.725924 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722750 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 02:39:36.725924 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722755 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 02:39:36.725924 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722759 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 02:39:36.725924 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722763 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 02:39:36.725924 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722767 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 02:39:36.725924 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722771 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 02:39:36.725924 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722776 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 02:39:36.725924 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722779 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 02:39:36.725924 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722783 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 02:39:36.725924 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722790 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 02:39:36.725924 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722796 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 02:39:36.725924 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722801 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 02:39:36.725924 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722808 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 02:39:36.725924 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722812 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 02:39:36.725924 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722817 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 02:39:36.725924 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722821 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 02:39:36.725924 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722825 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 02:39:36.725924 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722829 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 02:39:36.725924 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722835 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 02:39:36.726426 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722840 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 02:39:36.726426 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722844 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 02:39:36.726426 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722848 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 02:39:36.726426 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722852 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 02:39:36.726426 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722856 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 02:39:36.726426 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722861 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 02:39:36.726426 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722864 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 02:39:36.726426 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722869 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 02:39:36.726426 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722873 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 02:39:36.726426 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722877 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 02:39:36.726426 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722881 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 02:39:36.726426 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722885 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 02:39:36.726426 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722891 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 02:39:36.726426 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722895 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 02:39:36.726426 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722901 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 02:39:36.726426 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722906 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 02:39:36.726426 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722910 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 02:39:36.726426 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722914 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 02:39:36.726426 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722919 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 18 02:39:36.726945 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722923 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 02:39:36.726945 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722927 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 02:39:36.726945 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722931 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 02:39:36.726945 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722934 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 02:39:36.726945 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722938 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 02:39:36.726945 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722942 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 02:39:36.726945 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722946 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 02:39:36.726945 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722951 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 02:39:36.726945 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722955 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 02:39:36.726945 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722958 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 02:39:36.726945 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722962 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 02:39:36.726945 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722966 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 02:39:36.726945 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722970 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 02:39:36.726945 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722975 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 02:39:36.726945 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722979 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 02:39:36.726945 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722984 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 02:39:36.726945 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722987 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 02:39:36.726945 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722991 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 02:39:36.726945 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722995 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 02:39:36.727455 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.722999 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 02:39:36.727455 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723003 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 02:39:36.727455 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723007 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 02:39:36.727455 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723011 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 02:39:36.727455 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723015 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 02:39:36.727455 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723020 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 02:39:36.727455 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723025 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 02:39:36.727455 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723029 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 02:39:36.727455 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723033 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 02:39:36.727455 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723037 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 02:39:36.727455 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723041 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 02:39:36.727455 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723048 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 02:39:36.727455 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723052 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 02:39:36.727455 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723057 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 02:39:36.727455 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723061 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 02:39:36.727455 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723066 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 02:39:36.727455 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723070 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 02:39:36.727455 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723074 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 02:39:36.727455 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723078 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 02:39:36.727455 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723082 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 02:39:36.727919 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723086 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 02:39:36.727919 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723090 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 02:39:36.727919 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723681 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 02:39:36.727919 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723690 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 02:39:36.727919 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723696 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 02:39:36.727919 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723700 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 02:39:36.727919 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723710 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 02:39:36.727919 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723718 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 02:39:36.727919 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723724 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 02:39:36.727919 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723729 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 02:39:36.727919 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723733 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 02:39:36.727919 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723738 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 02:39:36.727919 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723742 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 02:39:36.727919 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723746 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 02:39:36.727919 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723751 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 02:39:36.727919 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723755 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 02:39:36.727919 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723759 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 02:39:36.727919 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723763 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 02:39:36.727919 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723768 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 02:39:36.728379 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723773 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 02:39:36.728379 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723777 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 02:39:36.728379 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723781 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 02:39:36.728379 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723785 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 02:39:36.728379 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723789 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 02:39:36.728379 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723793 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 02:39:36.728379 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723797 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 02:39:36.728379 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723801 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 02:39:36.728379 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723805 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 02:39:36.728379 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723809 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 02:39:36.728379 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723814 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 02:39:36.728379 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723817 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 02:39:36.728379 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723821 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 02:39:36.728379 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723826 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 02:39:36.728379 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723830 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 02:39:36.728379 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723834 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 02:39:36.728379 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723838 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 02:39:36.728379 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723841 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 02:39:36.728379 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723845 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 02:39:36.728379 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723850 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 02:39:36.728379 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723855 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 02:39:36.729072 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723859 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 02:39:36.729072 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723864 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 02:39:36.729072 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723869 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 02:39:36.729072 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723873 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 02:39:36.729072 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723877 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 02:39:36.729072 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723881 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 02:39:36.729072 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723885 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 02:39:36.729072 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723889 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 02:39:36.729072 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723893 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 18 02:39:36.729072 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723897 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 02:39:36.729072 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723902 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 02:39:36.729072 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723905 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 02:39:36.729072 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723910 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 02:39:36.729072 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723914 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 02:39:36.729072 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723918 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 02:39:36.729072 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723922 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 02:39:36.729072 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723926 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 02:39:36.729072 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723930 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 02:39:36.729072 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723934 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 02:39:36.729072 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723938 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 02:39:36.729567 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723942 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 02:39:36.729567 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723948 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 02:39:36.729567 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723954 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 02:39:36.729567 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723959 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 02:39:36.729567 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723963 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 02:39:36.729567 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723967 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 02:39:36.729567 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723971 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 02:39:36.729567 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723976 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 02:39:36.729567 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723980 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 02:39:36.729567 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723985 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 02:39:36.729567 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723989 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 02:39:36.729567 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723994 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 02:39:36.729567 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.723999 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 02:39:36.729567 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.724003 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 02:39:36.729567 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.724007 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 02:39:36.729567 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.724011 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 02:39:36.729567 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.724015 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 02:39:36.729567 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.724019 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 02:39:36.729567 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.724023 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 02:39:36.730075 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.724027 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 02:39:36.730075 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.724032 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 02:39:36.730075 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.724036 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 02:39:36.730075 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.724040 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 02:39:36.730075 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.724043 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 02:39:36.730075 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.724047 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 02:39:36.730075 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.724052 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 02:39:36.730075 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.724057 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 02:39:36.730075 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.724061 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 02:39:36.730075 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.724065 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 02:39:36.730075 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.724069 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 02:39:36.730075 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.724073 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 02:39:36.730075 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.724078 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 02:39:36.730075 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.724082 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 02:39:36.730075 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.724086 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 02:39:36.730075 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.724090 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 02:39:36.730075 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.724094 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 02:39:36.730075 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.724099 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 02:39:36.730075 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.724105 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 02:39:36.730592 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.724109 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 02:39:36.730592 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.724114 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 02:39:36.730592 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.724118 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 02:39:36.730592 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724219 2585 flags.go:64] FLAG: --address="0.0.0.0" Jul 18 02:39:36.730592 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724238 2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 18 02:39:36.730592 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724247 2585 flags.go:64] FLAG: --anonymous-auth="true" Jul 18 02:39:36.730592 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724253 2585 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 18 02:39:36.730592 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724267 2585 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 18 02:39:36.730592 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724273 2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 18 02:39:36.730592 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724279 2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 18 02:39:36.730592 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724285 2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 18 02:39:36.730592 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724290 2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 18 02:39:36.730592 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724295 2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 18 02:39:36.730592 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724300 2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 18 02:39:36.730592 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724305 2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 18 02:39:36.730592 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724310 2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 18 02:39:36.730592 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724314 2585 flags.go:64] FLAG: --cgroup-root="" Jul 18 02:39:36.730592 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724319 2585 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 18 02:39:36.730592 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724324 2585 flags.go:64] FLAG: --client-ca-file="" Jul 18 02:39:36.730592 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724329 2585 flags.go:64] FLAG: --cloud-provider="external" Jul 18 02:39:36.730592 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724333 2585 flags.go:64] FLAG: --cluster-dns="[]" Jul 18 02:39:36.730592 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724343 2585 flags.go:64] FLAG: --cluster-domain="" Jul 18 02:39:36.730592 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724348 2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 18 02:39:36.730592 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724352 2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 18 02:39:36.731167 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724359 2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 18 02:39:36.731167 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724365 2585 flags.go:64] FLAG: --container-log-max-files="5" Jul 18 02:39:36.731167 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724371 2585 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 18 02:39:36.731167 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724376 2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 18 02:39:36.731167 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724380 2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 18 02:39:36.731167 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724386 2585 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 18 02:39:36.731167 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724390 2585 flags.go:64] FLAG: --contention-profiling="false" Jul 18 02:39:36.731167 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724394 2585 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 18 02:39:36.731167 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724399 2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 18 02:39:36.731167 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724404 2585 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 18 02:39:36.731167 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724408 2585 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 18 02:39:36.731167 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724414 2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 18 02:39:36.731167 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724420 2585 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 18 02:39:36.731167 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724425 2585 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 18 02:39:36.731167 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724429 2585 flags.go:64] FLAG: --enable-load-reader="false" Jul 18 02:39:36.731167 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724433 2585 flags.go:64] FLAG: --enable-server="true" Jul 18 02:39:36.731167 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724463 2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 18 02:39:36.731167 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724479 2585 flags.go:64] FLAG: --event-burst="100" Jul 18 02:39:36.731167 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724484 2585 flags.go:64] FLAG: --event-qps="50" Jul 18 02:39:36.731167 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724488 2585 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 18 02:39:36.731167 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724493 2585 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 18 02:39:36.731167 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724497 2585 flags.go:64] FLAG: --eviction-hard="" Jul 18 02:39:36.731167 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724503 2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 18 02:39:36.731167 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724508 2585 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 18 02:39:36.731747 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724512 2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 18 02:39:36.731747 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724517 2585 flags.go:64] FLAG: --eviction-soft="" Jul 18 02:39:36.731747 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724522 2585 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 18 02:39:36.731747 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724527 2585 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 18 02:39:36.731747 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724531 2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 18 02:39:36.731747 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724535 2585 flags.go:64] FLAG: --experimental-mounter-path="" Jul 18 02:39:36.731747 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724540 2585 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 18 02:39:36.731747 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724545 2585 flags.go:64] FLAG: --fail-swap-on="true" Jul 18 02:39:36.731747 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724551 2585 flags.go:64] FLAG: --feature-gates="" Jul 18 02:39:36.731747 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724557 2585 flags.go:64] FLAG: --file-check-frequency="20s" Jul 18 02:39:36.731747 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724562 2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 18 02:39:36.731747 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724567 2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 18 02:39:36.731747 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724572 2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 18 02:39:36.731747 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724576 2585 flags.go:64] FLAG: --healthz-port="10248" Jul 18 02:39:36.731747 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724581 2585 flags.go:64] FLAG: --help="false" Jul 18 02:39:36.731747 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724585 2585 flags.go:64] FLAG: --hostname-override="ip-10-0-140-4.ec2.internal" Jul 18 02:39:36.731747 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724590 2585 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 18 02:39:36.731747 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724595 2585 flags.go:64] FLAG: --http-check-frequency="20s" Jul 18 02:39:36.731747 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724600 2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 18 02:39:36.731747 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724605 2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 18 02:39:36.731747 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724611 2585 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 18 02:39:36.731747 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724616 2585 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 18 02:39:36.731747 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724621 2585 flags.go:64] FLAG: --image-service-endpoint="" Jul 18 02:39:36.732320 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724625 2585 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 18 02:39:36.732320 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724630 2585 flags.go:64] FLAG: --kube-api-burst="100" Jul 18 02:39:36.732320 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724644 2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 18 02:39:36.732320 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724649 2585 flags.go:64] FLAG: --kube-api-qps="50" Jul 18 02:39:36.732320 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724654 2585 flags.go:64] FLAG: --kube-reserved="" Jul 18 02:39:36.732320 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724658 2585 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 18 02:39:36.732320 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724663 2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 18 02:39:36.732320 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724667 2585 flags.go:64] FLAG: --kubelet-cgroups="" Jul 18 02:39:36.732320 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724672 2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 18 02:39:36.732320 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724676 2585 flags.go:64] FLAG: --lock-file="" Jul 18 02:39:36.732320 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724681 2585 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 18 02:39:36.732320 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724685 2585 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 18 02:39:36.732320 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724689 2585 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 18 02:39:36.732320 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724698 2585 flags.go:64] FLAG: --log-json-split-stream="false" Jul 18 02:39:36.732320 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724703 2585 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 18 02:39:36.732320 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724707 2585 flags.go:64] FLAG: --log-text-split-stream="false" Jul 18 02:39:36.732320 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724712 2585 flags.go:64] FLAG: --logging-format="text" Jul 18 02:39:36.732320 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724718 2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 18 02:39:36.732320 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724723 2585 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 18 02:39:36.732320 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724727 2585 flags.go:64] FLAG: --manifest-url="" Jul 18 02:39:36.732320 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724732 2585 flags.go:64] FLAG: --manifest-url-header="" Jul 18 02:39:36.732320 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724738 2585 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 18 02:39:36.732320 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724743 2585 flags.go:64] FLAG: --max-open-files="1000000" Jul 18 02:39:36.732320 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724749 2585 flags.go:64] FLAG: --max-pods="110" Jul 18 02:39:36.732320 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724753 2585 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 18 02:39:36.732894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724758 2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 18 02:39:36.732894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724762 2585 flags.go:64] FLAG: --memory-manager-policy="None" Jul 18 02:39:36.732894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724766 2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 18 02:39:36.732894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724771 2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 18 02:39:36.732894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724778 2585 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 18 02:39:36.732894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724782 2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 18 02:39:36.732894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724793 2585 flags.go:64] FLAG: --node-status-max-images="50" Jul 18 02:39:36.732894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724797 2585 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 18 02:39:36.732894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724802 2585 flags.go:64] FLAG: --oom-score-adj="-999" Jul 18 02:39:36.732894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724806 2585 flags.go:64] FLAG: --pod-cidr="" Jul 18 02:39:36.732894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724811 2585 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 18 02:39:36.732894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724819 2585 flags.go:64] FLAG: --pod-manifest-path="" Jul 18 02:39:36.732894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724823 2585 flags.go:64] FLAG: --pod-max-pids="-1" Jul 18 02:39:36.732894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724828 2585 flags.go:64] FLAG: --pods-per-core="0" Jul 18 02:39:36.732894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724832 2585 flags.go:64] FLAG: --port="10250" Jul 18 02:39:36.732894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724837 2585 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 18 02:39:36.732894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724841 2585 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-072098526d64bf671" Jul 18 02:39:36.732894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724846 2585 flags.go:64] FLAG: --qos-reserved="" Jul 18 02:39:36.732894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724851 2585 flags.go:64] FLAG: --read-only-port="10255" Jul 18 02:39:36.732894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724856 2585 flags.go:64] FLAG: --register-node="true" Jul 18 02:39:36.732894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724860 2585 flags.go:64] FLAG: --register-with-taints="" Jul 18 02:39:36.732894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724866 2585 flags.go:64] FLAG: --registry-burst="10" Jul 18 02:39:36.732894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724870 2585 flags.go:64] FLAG: --registry-qps="5" Jul 18 02:39:36.732894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724874 2585 flags.go:64] FLAG: --reserved-cpus="" Jul 18 02:39:36.733544 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724880 2585 flags.go:64] FLAG: --reserved-memory="" Jul 18 02:39:36.733544 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724886 2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 18 02:39:36.733544 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724892 2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 18 02:39:36.733544 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724896 2585 flags.go:64] FLAG: --rotate-certificates="false" Jul 18 02:39:36.733544 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724901 2585 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 18 02:39:36.733544 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724905 2585 flags.go:64] FLAG: --runonce="false" Jul 18 02:39:36.733544 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724909 2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 18 02:39:36.733544 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724914 2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 18 02:39:36.733544 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724919 2585 flags.go:64] FLAG: --seccomp-default="false" Jul 18 02:39:36.733544 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724924 2585 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 18 02:39:36.733544 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724928 2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 18 02:39:36.733544 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724932 2585 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 18 02:39:36.733544 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724937 2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 18 02:39:36.733544 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724943 2585 flags.go:64] FLAG: --storage-driver-password="root" Jul 18 02:39:36.733544 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724947 2585 flags.go:64] FLAG: --storage-driver-secure="false" Jul 18 02:39:36.733544 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724952 2585 flags.go:64] FLAG: --storage-driver-table="stats" Jul 18 02:39:36.733544 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724956 2585 flags.go:64] FLAG: --storage-driver-user="root" Jul 18 02:39:36.733544 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724960 2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 18 02:39:36.733544 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724965 2585 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 18 02:39:36.733544 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724970 2585 flags.go:64] FLAG: --system-cgroups="" Jul 18 02:39:36.733544 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724975 2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 18 02:39:36.733544 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724983 2585 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 18 02:39:36.733544 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724987 2585 flags.go:64] FLAG: --tls-cert-file="" Jul 18 02:39:36.733544 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724991 2585 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 18 02:39:36.733544 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.724998 2585 flags.go:64] FLAG: --tls-min-version="" Jul 18 02:39:36.734132 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.725002 2585 flags.go:64] FLAG: --tls-private-key-file="" Jul 18 02:39:36.734132 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.725007 2585 flags.go:64] FLAG: --topology-manager-policy="none" Jul 18 02:39:36.734132 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.725011 2585 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 18 02:39:36.734132 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.725016 2585 flags.go:64] FLAG: --topology-manager-scope="container" Jul 18 02:39:36.734132 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.725020 2585 flags.go:64] FLAG: --v="2" Jul 18 02:39:36.734132 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.725026 2585 flags.go:64] FLAG: --version="false" Jul 18 02:39:36.734132 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.725032 2585 flags.go:64] FLAG: --vmodule="" Jul 18 02:39:36.734132 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.725039 2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 18 02:39:36.734132 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.725045 2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 18 02:39:36.734132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725237 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 02:39:36.734132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725244 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 02:39:36.734132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725248 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 02:39:36.734132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725253 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 02:39:36.734132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725257 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 02:39:36.734132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725261 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 02:39:36.734132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725265 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 02:39:36.734132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725269 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 02:39:36.734132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725273 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 02:39:36.734132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725277 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 02:39:36.734132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725281 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 02:39:36.734132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725287 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 02:39:36.734132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725292 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 18 02:39:36.734661 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725296 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 02:39:36.734661 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725300 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 02:39:36.734661 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725304 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 02:39:36.734661 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725308 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 02:39:36.734661 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725312 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 02:39:36.734661 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725316 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 02:39:36.734661 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725320 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 02:39:36.734661 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725325 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 02:39:36.734661 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725329 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 02:39:36.734661 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725333 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 02:39:36.734661 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725337 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 02:39:36.734661 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725341 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 02:39:36.734661 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725345 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 02:39:36.734661 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725349 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 02:39:36.734661 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725353 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 02:39:36.734661 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725357 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 02:39:36.734661 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725362 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 02:39:36.734661 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725368 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 02:39:36.734661 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725372 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 02:39:36.734661 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725376 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 02:39:36.735132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725380 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 02:39:36.735132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725384 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 02:39:36.735132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725388 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 02:39:36.735132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725392 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 02:39:36.735132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725395 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 02:39:36.735132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725400 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 02:39:36.735132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725403 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 02:39:36.735132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725408 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 02:39:36.735132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725412 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 02:39:36.735132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725416 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 02:39:36.735132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725421 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 02:39:36.735132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725425 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 02:39:36.735132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725429 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 02:39:36.735132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725434 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 02:39:36.735132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725438 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 02:39:36.735132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725441 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 02:39:36.735132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725445 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 02:39:36.735132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725450 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 02:39:36.735132 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725461 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 02:39:36.735595 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725467 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 02:39:36.735595 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725472 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 02:39:36.735595 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725476 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 02:39:36.735595 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725480 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 02:39:36.735595 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725485 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 02:39:36.735595 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725491 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 02:39:36.735595 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725496 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 02:39:36.735595 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725501 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 02:39:36.735595 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725505 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 02:39:36.735595 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725510 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 02:39:36.735595 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725517 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 02:39:36.735595 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725521 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 02:39:36.735595 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725525 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 02:39:36.735595 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725530 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 02:39:36.735595 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725534 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 02:39:36.735595 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725539 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 02:39:36.735595 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725543 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 02:39:36.735595 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725547 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 02:39:36.735595 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725551 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 02:39:36.736046 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725555 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 02:39:36.736046 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725562 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 02:39:36.736046 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725567 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 02:39:36.736046 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725572 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 02:39:36.736046 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725577 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 02:39:36.736046 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725582 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 02:39:36.736046 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725586 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 02:39:36.736046 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725590 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 02:39:36.736046 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725594 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 02:39:36.736046 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725598 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 02:39:36.736046 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725602 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 02:39:36.736046 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725606 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 02:39:36.736046 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725610 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 02:39:36.736046 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725615 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 02:39:36.736046 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725619 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 02:39:36.736046 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725623 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 02:39:36.736046 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725627 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 02:39:36.736046 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725631 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 02:39:36.736046 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725635 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 02:39:36.736503 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725639 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 02:39:36.736503 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725643 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 02:39:36.736503 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725647 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 02:39:36.736503 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725651 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 02:39:36.736503 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725657 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 02:39:36.736503 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725661 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 02:39:36.736503 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725665 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 02:39:36.736503 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725669 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 02:39:36.736503 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.725673 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 02:39:36.736503 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.726620 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 18 02:39:36.736503 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.734823 2585 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 18 02:39:36.736503 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.734839 2585 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 18 02:39:36.736503 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.735711 2585 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 18 02:39:36.736503 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.735719 2585 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 18 02:39:36.736503 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735791 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 02:39:36.736865 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735797 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 02:39:36.736865 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735800 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 02:39:36.736865 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735802 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 02:39:36.736865 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735805 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 02:39:36.736865 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735808 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 02:39:36.736865 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735811 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 02:39:36.736865 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735814 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 02:39:36.736865 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735816 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 02:39:36.736865 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735819 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 02:39:36.736865 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735821 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 02:39:36.736865 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735824 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 02:39:36.736865 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735827 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 02:39:36.736865 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735829 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 02:39:36.736865 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735831 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 02:39:36.736865 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735834 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 02:39:36.736865 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735836 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 02:39:36.736865 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735839 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 02:39:36.736865 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735841 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 02:39:36.736865 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735844 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 02:39:36.736865 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735847 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 02:39:36.737366 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735849 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 02:39:36.737366 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735852 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 02:39:36.737366 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735854 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 02:39:36.737366 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735856 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 02:39:36.737366 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735859 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 02:39:36.737366 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735861 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 02:39:36.737366 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735864 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 02:39:36.737366 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735866 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 02:39:36.737366 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735877 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 02:39:36.737366 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735880 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 02:39:36.737366 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735882 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 02:39:36.737366 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735885 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 02:39:36.737366 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735888 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 02:39:36.737366 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735890 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 02:39:36.737366 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735892 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 02:39:36.737366 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735895 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 02:39:36.737366 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735898 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 02:39:36.737366 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735900 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 02:39:36.737366 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735902 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 02:39:36.737366 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735905 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 02:39:36.737830 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735907 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 02:39:36.737830 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735909 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 02:39:36.737830 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735912 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 02:39:36.737830 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735914 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 02:39:36.737830 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735916 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 02:39:36.737830 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735919 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 02:39:36.737830 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735921 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 02:39:36.737830 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735924 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 02:39:36.737830 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735926 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 02:39:36.737830 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735929 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 02:39:36.737830 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735931 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 02:39:36.737830 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735933 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 02:39:36.737830 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735936 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 02:39:36.737830 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735938 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 02:39:36.737830 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735941 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 02:39:36.737830 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735943 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 02:39:36.737830 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735946 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 02:39:36.737830 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735948 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 02:39:36.737830 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735951 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 02:39:36.737830 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735954 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 02:39:36.738506 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735958 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 02:39:36.738506 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735967 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 02:39:36.738506 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735970 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 02:39:36.738506 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735974 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 02:39:36.738506 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735978 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 02:39:36.738506 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735982 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 02:39:36.738506 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735985 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 02:39:36.738506 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735987 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 02:39:36.738506 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735990 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 18 02:39:36.738506 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735993 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 02:39:36.738506 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735995 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 02:39:36.738506 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.735998 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 02:39:36.738506 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736000 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 02:39:36.738506 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736003 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 02:39:36.738506 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736005 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 02:39:36.738506 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736007 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 02:39:36.738506 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736010 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 02:39:36.738506 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736012 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 02:39:36.738506 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736015 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 02:39:36.739257 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736017 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 02:39:36.739257 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736019 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 02:39:36.739257 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736022 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 02:39:36.739257 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736024 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 02:39:36.739257 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736027 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 02:39:36.739257 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736029 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 02:39:36.739257 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736031 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 02:39:36.739257 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736034 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 02:39:36.739257 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736036 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 02:39:36.739257 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736038 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 02:39:36.739257 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736041 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 02:39:36.739257 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736044 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 02:39:36.739257 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736046 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 02:39:36.739257 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736049 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 02:39:36.739257 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736052 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 02:39:36.739257 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736060 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 02:39:36.739257 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736063 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 02:39:36.739257 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736065 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 02:39:36.739257 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736068 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 02:39:36.739745 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.736072 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 18 02:39:36.739745 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736240 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 02:39:36.739745 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736245 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 02:39:36.739745 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736248 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 02:39:36.739745 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736250 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 02:39:36.739745 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736253 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 02:39:36.739745 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736256 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 02:39:36.739745 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736258 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 02:39:36.739745 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736260 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 02:39:36.739745 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736263 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 02:39:36.739745 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736265 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 02:39:36.739745 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736268 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 02:39:36.739745 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736270 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 02:39:36.739745 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736273 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 02:39:36.739745 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736275 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 02:39:36.740101 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736278 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 18 02:39:36.740101 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736280 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 02:39:36.740101 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736283 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 02:39:36.740101 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736285 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 02:39:36.740101 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736287 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 02:39:36.740101 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736290 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 02:39:36.740101 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736294 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 02:39:36.740101 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736297 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 02:39:36.740101 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736300 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 02:39:36.740101 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736302 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 02:39:36.740101 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736306 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 02:39:36.740101 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736308 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 02:39:36.740101 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736311 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 02:39:36.740101 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736313 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 02:39:36.740101 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736321 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 02:39:36.740101 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736324 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 02:39:36.740101 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736326 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 02:39:36.740101 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736329 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 02:39:36.740101 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736331 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 02:39:36.740101 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736333 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 02:39:36.740605 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736336 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 02:39:36.740605 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736338 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 02:39:36.740605 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736340 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 02:39:36.740605 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736343 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 02:39:36.740605 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736345 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 02:39:36.740605 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736347 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 02:39:36.740605 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736350 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 02:39:36.740605 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736352 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 02:39:36.740605 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736355 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 02:39:36.740605 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736357 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 02:39:36.740605 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736360 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 02:39:36.740605 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736362 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 02:39:36.740605 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736364 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 02:39:36.740605 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736367 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 02:39:36.740605 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736369 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 02:39:36.740605 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736372 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 02:39:36.740605 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736374 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 02:39:36.740605 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736376 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 02:39:36.740605 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736379 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 02:39:36.740605 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736381 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 02:39:36.741068 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736383 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 02:39:36.741068 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736386 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 02:39:36.741068 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736388 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 02:39:36.741068 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736391 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 02:39:36.741068 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736393 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 02:39:36.741068 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736395 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 02:39:36.741068 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736398 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 02:39:36.741068 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736406 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 02:39:36.741068 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736409 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 02:39:36.741068 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736411 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 02:39:36.741068 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736413 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 02:39:36.741068 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736416 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 02:39:36.741068 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736418 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 02:39:36.741068 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736421 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 02:39:36.741068 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736423 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 02:39:36.741068 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736425 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 02:39:36.741068 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736427 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 02:39:36.741068 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736430 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 02:39:36.741068 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736433 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 02:39:36.741068 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736436 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 02:39:36.741562 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736439 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 02:39:36.741562 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736441 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 02:39:36.741562 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736443 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 02:39:36.741562 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736446 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 02:39:36.741562 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736448 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 02:39:36.741562 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736451 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 02:39:36.741562 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736453 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 02:39:36.741562 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736455 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 02:39:36.741562 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736458 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 02:39:36.741562 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736460 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 02:39:36.741562 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736462 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 02:39:36.741562 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736465 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 02:39:36.741562 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736467 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 02:39:36.741562 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736470 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 02:39:36.741562 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736472 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 02:39:36.741562 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736475 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 02:39:36.741562 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736477 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 02:39:36.741562 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736480 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 02:39:36.741562 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736482 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 02:39:36.741562 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736484 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 02:39:36.742016 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736494 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 02:39:36.742016 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736497 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 02:39:36.742016 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736500 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 02:39:36.742016 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736503 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 02:39:36.742016 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:36.736506 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 02:39:36.742016 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.736511 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 18 02:39:36.742016 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.736627 2585 server.go:962] "Client rotation is on, will bootstrap in background" Jul 18 02:39:36.742016 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.740516 2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 18 02:39:36.742308 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.742293 2585 server.go:1019] "Starting client certificate rotation" Jul 18 02:39:36.742405 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.742390 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 18 02:39:36.742439 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.742431 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 18 02:39:36.745004 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.744992 2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 18 02:39:36.749017 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.748998 2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 18 02:39:36.757575 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.757557 2585 log.go:25] "Validated CRI v1 runtime API" Jul 18 02:39:36.760252 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.760238 2585 log.go:25] "Validated CRI v1 image API" Jul 18 02:39:36.761012 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.760999 2585 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 18 02:39:36.763449 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.763432 2585 fs.go:135] Filesystem UUIDs: map[0f8a7f14-2256-4756-adec-9ef5ced22611:/dev/nvme0n1p4 2c2de8a0-9796-40c8-9f18-f6b33a3b6036:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 18 02:39:36.763505 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.763449 2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 18 02:39:36.769759 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.769640 2585 manager.go:216] Machine: {Timestamp:2026-07-18 02:39:36.767425599 +0000 UTC m=+0.371073276 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3204050 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2138f71fdbe2eff867dcffeb60bf34 SystemUUID:ec2138f7-1fdb-e2ef-f867-dcffeb60bf34 BootID:f87717fa-f571-4908-8785-e41d3cbd5025 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ca:60:82:3a:e7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ca:60:82:3a:e7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:5f:6e:ce:00:d6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 18 02:39:36.769759 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.769757 2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 18 02:39:36.769864 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.769822 2585 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 18 02:39:36.772568 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.772542 2585 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 18 02:39:36.772723 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.772574 2585 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-4.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 18 02:39:36.773436 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.773425 2585 topology_manager.go:138] "Creating topology manager with none policy" Jul 18 02:39:36.773487 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.773440 2585 container_manager_linux.go:309] "Creating device plugin manager" Jul 18 02:39:36.773487 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.773449 2585 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 18 02:39:36.773487 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.773467 2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 18 02:39:36.775327 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.775314 2585 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 18 02:39:36.778232 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.778163 2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 18 02:39:36.780520 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.780507 2585 state_mem.go:36] "Initialized new in-memory state store" Jul 18 02:39:36.780657 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.780646 2585 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 18 02:39:36.780709 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.780682 2585 kubelet.go:486] "Attempting to sync node with API server" Jul 18 02:39:36.780709 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.780696 2585 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 18 02:39:36.780789 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.780711 2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 18 02:39:36.780789 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.780723 2585 kubelet.go:398] "Adding apiserver pod source" Jul 18 02:39:36.780789 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.780736 2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 18 02:39:36.781255 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.781230 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 18 02:39:36.781336 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.781273 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 18 02:39:36.786030 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.786005 2585 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 18 02:39:36.788771 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.788756 2585 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 18 02:39:36.788847 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.788787 2585 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 18 02:39:36.789671 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.789660 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 18 02:39:36.789712 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.789675 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 18 02:39:36.789712 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.789681 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 18 02:39:36.789712 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.789686 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 18 02:39:36.789712 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.789694 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 18 02:39:36.789712 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.789699 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 18 02:39:36.789712 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.789706 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 18 02:39:36.789712 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.789711 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 18 02:39:36.789712 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.789716 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 18 02:39:36.789930 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.789722 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 18 02:39:36.789930 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.789730 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 18 02:39:36.789930 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.789735 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 18 02:39:36.790603 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.790589 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 18 02:39:36.790603 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.790600 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 18 02:39:36.792809 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.792791 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-4.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 18 02:39:36.793488 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.793476 2585 server.go:1268] "Started kubelet" Jul 18 02:39:36.793592 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.793571 2585 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 18 02:39:36.793654 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.793575 2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 18 02:39:36.793654 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.793643 2585 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 18 02:39:36.793902 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:36.793880 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 18 02:39:36.793998 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:36.793883 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-4.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 18 02:39:36.794262 ip-10-0-140-4 systemd[1]: Started Kubernetes Kubelet. Jul 18 02:39:36.795207 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.795165 2585 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 18 02:39:36.795389 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.795375 2585 server.go:310] "Adding debug handlers to kubelet server" Jul 18 02:39:36.800751 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:36.800737 2585 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 18 02:39:36.804669 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:36.803547 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-4.ec2.internal.18c341f01673316c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-4.ec2.internal,UID:ip-10-0-140-4.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-4.ec2.internal,},FirstTimestamp:2026-07-18 02:39:36.793452908 +0000 UTC m=+0.397100585,LastTimestamp:2026-07-18 02:39:36.793452908 +0000 UTC m=+0.397100585,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-4.ec2.internal,}" Jul 18 02:39:36.804831 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.804713 2585 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 18 02:39:36.804831 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.804725 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 18 02:39:36.804831 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:36.804778 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-4.ec2.internal\" not found" Jul 18 02:39:36.805476 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.805462 2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 18 02:39:36.805560 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.805469 2585 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 18 02:39:36.805560 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.805509 2585 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 18 02:39:36.805661 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.805561 2585 factory.go:55] Registering systemd factory Jul 18 02:39:36.805661 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.805576 2585 factory.go:223] Registration of the systemd container factory successfully Jul 18 02:39:36.805749 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.805702 2585 reconstruct.go:97] "Volume reconstruction finished" Jul 18 02:39:36.805749 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.805713 2585 reconciler.go:29] "Reconciler: start to sync state" Jul 18 02:39:36.805838 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.805803 2585 factory.go:153] Registering CRI-O factory Jul 18 02:39:36.805838 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.805817 2585 factory.go:223] Registration of the crio container factory successfully Jul 18 02:39:36.805906 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.805867 2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 18 02:39:36.805906 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.805885 2585 factory.go:103] Registering Raw factory Jul 18 02:39:36.805906 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.805896 2585 manager.go:1196] Started watching for new ooms in manager Jul 18 02:39:36.806267 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.806247 2585 manager.go:349] Starting recovery of all containers Jul 18 02:39:36.817534 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:36.817501 2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-4.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 18 02:39:36.817718 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:36.817693 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 18 02:39:36.818025 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.818001 2585 manager.go:354] Recovery completed Jul 18 02:39:36.823550 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.823537 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 02:39:36.825313 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.825295 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-4.ec2.internal" event="NodeHasSufficientMemory" Jul 18 02:39:36.825397 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.825323 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-4.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 02:39:36.825397 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.825334 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-4.ec2.internal" event="NodeHasSufficientPID" Jul 18 02:39:36.825891 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.825874 2585 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 18 02:39:36.825891 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.825888 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 02:39:36.826026 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.825890 2585 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 18 02:39:36.826026 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.825942 2585 state_mem.go:36] "Initialized new in-memory state store" Jul 18 02:39:36.826617 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.826596 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-4.ec2.internal" event="NodeHasSufficientMemory" Jul 18 02:39:36.826698 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.826624 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-4.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 02:39:36.826698 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.826633 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-4.ec2.internal" event="NodeHasSufficientPID" Jul 18 02:39:36.828363 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.828350 2585 policy_none.go:49] "None policy: Start" Jul 18 02:39:36.828363 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.828365 2585 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 18 02:39:36.828507 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.828375 2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 18 02:39:36.830306 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.830294 2585 policy_none.go:47] "Start" Jul 18 02:39:36.833271 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:36.833176 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-4.ec2.internal.18c341f018595483 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-4.ec2.internal,UID:ip-10-0-140-4.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-4.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-4.ec2.internal,},FirstTimestamp:2026-07-18 02:39:36.825312387 +0000 UTC m=+0.428960066,LastTimestamp:2026-07-18 02:39:36.825312387 +0000 UTC m=+0.428960066,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-4.ec2.internal,}" Jul 18 02:39:36.837210 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:36.837107 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-4.ec2.internal.18c341f0185992c7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-4.ec2.internal,UID:ip-10-0-140-4.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-4.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-4.ec2.internal,},FirstTimestamp:2026-07-18 02:39:36.825328327 +0000 UTC m=+0.428976004,LastTimestamp:2026-07-18 02:39:36.825328327 +0000 UTC m=+0.428976004,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-4.ec2.internal,}" Jul 18 02:39:36.840861 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:36.840735 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-4.ec2.internal.18c341f01859b7f1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-4.ec2.internal,UID:ip-10-0-140-4.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-4.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-4.ec2.internal,},FirstTimestamp:2026-07-18 02:39:36.825337841 +0000 UTC m=+0.428985521,LastTimestamp:2026-07-18 02:39:36.825337841 +0000 UTC m=+0.428985521,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-4.ec2.internal,}" Jul 18 02:39:36.843568 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.843551 2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xtnxf" Jul 18 02:39:36.848015 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:36.847945 2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-4.ec2.internal.18c341f018595483\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-4.ec2.internal.18c341f018595483 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-4.ec2.internal,UID:ip-10-0-140-4.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-4.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-4.ec2.internal,},FirstTimestamp:2026-07-18 02:39:36.825312387 +0000 UTC m=+0.428960066,LastTimestamp:2026-07-18 02:39:36.826613041 +0000 UTC m=+0.430260718,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-4.ec2.internal,}" Jul 18 02:39:36.874046 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:36.851307 2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-4.ec2.internal.18c341f0185992c7\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-4.ec2.internal.18c341f0185992c7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-4.ec2.internal,UID:ip-10-0-140-4.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-4.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-4.ec2.internal,},FirstTimestamp:2026-07-18 02:39:36.825328327 +0000 UTC m=+0.428976004,LastTimestamp:2026-07-18 02:39:36.826628746 +0000 UTC m=+0.430276422,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-4.ec2.internal,}" Jul 18 02:39:36.874046 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.851746 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xtnxf" Jul 18 02:39:36.874046 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:36.858978 2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-4.ec2.internal.18c341f01859b7f1\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-4.ec2.internal.18c341f01859b7f1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-4.ec2.internal,UID:ip-10-0-140-4.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-4.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-4.ec2.internal,},FirstTimestamp:2026-07-18 02:39:36.825337841 +0000 UTC m=+0.428985521,LastTimestamp:2026-07-18 02:39:36.826637238 +0000 UTC m=+0.430284917,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-4.ec2.internal,}" Jul 18 02:39:36.874046 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.860707 2585 manager.go:337] "Starting Device Plugin manager" Jul 18 02:39:36.874046 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:36.860729 2585 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 18 02:39:36.874046 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.860738 2585 server.go:87] "Starting device plugin registration server" Jul 18 02:39:36.874046 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.860929 2585 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 18 02:39:36.874046 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.860941 2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 18 02:39:36.874046 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.861046 2585 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 18 02:39:36.874046 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.861148 2585 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 18 02:39:36.874046 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.861158 2585 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 18 02:39:36.874046 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:36.861563 2585 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 18 02:39:36.874046 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:36.861608 2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-4.ec2.internal\" not found" Jul 18 02:39:36.892827 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.892808 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 18 02:39:36.893913 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.893881 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 18 02:39:36.893980 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.893933 2585 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 18 02:39:36.893980 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.893952 2585 kubelet.go:2448] "Starting kubelet main sync loop" Jul 18 02:39:36.894045 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:36.893989 2585 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 18 02:39:36.896392 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.896375 2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 02:39:36.961892 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.961870 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 02:39:36.962617 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.962604 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-4.ec2.internal" event="NodeHasSufficientMemory" Jul 18 02:39:36.962682 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.962629 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-4.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 02:39:36.962682 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.962644 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-4.ec2.internal" event="NodeHasSufficientPID" Jul 18 02:39:36.962682 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.962666 2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-4.ec2.internal" Jul 18 02:39:36.971747 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.971730 2585 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-4.ec2.internal" Jul 18 02:39:36.971747 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:36.971748 2585 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-4.ec2.internal\": node \"ip-10-0-140-4.ec2.internal\" not found" Jul 18 02:39:36.994346 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.994326 2585 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-4.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-4.ec2.internal"] Jul 18 02:39:36.994418 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.994376 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 02:39:36.995051 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.995035 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-4.ec2.internal" event="NodeHasSufficientMemory" Jul 18 02:39:36.995108 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.995062 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-4.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 02:39:36.995108 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.995072 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-4.ec2.internal" event="NodeHasSufficientPID" Jul 18 02:39:36.997408 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.997397 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 02:39:36.997544 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.997529 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-4.ec2.internal" Jul 18 02:39:36.997587 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.997560 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 02:39:36.998078 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.998054 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-4.ec2.internal" event="NodeHasSufficientMemory" Jul 18 02:39:36.998150 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.998069 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-4.ec2.internal" event="NodeHasSufficientMemory" Jul 18 02:39:36.998150 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.998087 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-4.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 02:39:36.998150 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.998101 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-4.ec2.internal" event="NodeHasSufficientPID" Jul 18 02:39:36.998150 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.998103 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-4.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 02:39:36.998150 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:36.998114 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-4.ec2.internal" event="NodeHasSufficientPID" Jul 18 02:39:36.999786 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:36.999770 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-4.ec2.internal\" not found" Jul 18 02:39:37.002065 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:37.002048 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-4.ec2.internal" Jul 18 02:39:37.002155 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:37.002079 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 02:39:37.002715 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:37.002700 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-4.ec2.internal" event="NodeHasSufficientMemory" Jul 18 02:39:37.002790 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:37.002728 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-4.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 02:39:37.002790 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:37.002741 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-4.ec2.internal" event="NodeHasSufficientPID" Jul 18 02:39:37.006998 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:37.006983 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea3598a841ad3d314209c397f6def29f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-4.ec2.internal\" (UID: \"ea3598a841ad3d314209c397f6def29f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-4.ec2.internal" Jul 18 02:39:37.007066 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:37.007006 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/10427b699bfe492315ce6d6f3f8955b6-config\") pod \"kube-apiserver-proxy-ip-10-0-140-4.ec2.internal\" (UID: \"10427b699bfe492315ce6d6f3f8955b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-4.ec2.internal" Jul 18 02:39:37.007066 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:37.007022 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ea3598a841ad3d314209c397f6def29f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-4.ec2.internal\" (UID: \"ea3598a841ad3d314209c397f6def29f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-4.ec2.internal" Jul 18 02:39:37.013682 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:37.013665 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-4.ec2.internal\" not found" node="ip-10-0-140-4.ec2.internal" Jul 18 02:39:37.018657 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:37.018644 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-4.ec2.internal\" not found" node="ip-10-0-140-4.ec2.internal" Jul 18 02:39:37.100739 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:37.100699 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-4.ec2.internal\" not found" Jul 18 02:39:37.108041 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:37.108021 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ea3598a841ad3d314209c397f6def29f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-4.ec2.internal\" (UID: \"ea3598a841ad3d314209c397f6def29f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-4.ec2.internal" Jul 18 02:39:37.108087 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:37.108044 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ea3598a841ad3d314209c397f6def29f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-4.ec2.internal\" (UID: \"ea3598a841ad3d314209c397f6def29f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-4.ec2.internal" Jul 18 02:39:37.108087 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:37.108055 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea3598a841ad3d314209c397f6def29f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-4.ec2.internal\" (UID: \"ea3598a841ad3d314209c397f6def29f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-4.ec2.internal" Jul 18 02:39:37.108150 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:37.108090 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/10427b699bfe492315ce6d6f3f8955b6-config\") pod \"kube-apiserver-proxy-ip-10-0-140-4.ec2.internal\" (UID: \"10427b699bfe492315ce6d6f3f8955b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-4.ec2.internal" Jul 18 02:39:37.108150 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:37.108122 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/10427b699bfe492315ce6d6f3f8955b6-config\") pod \"kube-apiserver-proxy-ip-10-0-140-4.ec2.internal\" (UID: \"10427b699bfe492315ce6d6f3f8955b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-4.ec2.internal" Jul 18 02:39:37.108150 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:37.108125 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea3598a841ad3d314209c397f6def29f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-4.ec2.internal\" (UID: \"ea3598a841ad3d314209c397f6def29f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-4.ec2.internal" Jul 18 02:39:37.200944 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:37.200918 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-4.ec2.internal\" not found" Jul 18 02:39:37.301703 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:37.301686 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-4.ec2.internal\" not found" Jul 18 02:39:37.315143 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:37.315127 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-4.ec2.internal" Jul 18 02:39:37.322489 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:37.322467 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-4.ec2.internal" Jul 18 02:39:37.402335 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:37.402285 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-4.ec2.internal\" not found" Jul 18 02:39:37.502816 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:37.502797 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-4.ec2.internal\" not found" Jul 18 02:39:37.603268 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:37.603249 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-4.ec2.internal\" not found" Jul 18 02:39:37.614911 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:37.614887 2585 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 02:39:37.704156 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:37.704099 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-4.ec2.internal\" not found" Jul 18 02:39:37.741782 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:37.741754 2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 18 02:39:37.742137 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:37.741915 2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 18 02:39:37.742137 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:37.741943 2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 18 02:39:37.804928 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:37.804900 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-4.ec2.internal\" not found" Jul 18 02:39:37.857249 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:37.857223 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10427b699bfe492315ce6d6f3f8955b6.slice/crio-ae6642774bd8b7cb9be5f78d2833985219e194cf07d50d7e0c8a1f417b539c9f WatchSource:0}: Error finding container ae6642774bd8b7cb9be5f78d2833985219e194cf07d50d7e0c8a1f417b539c9f: Status 404 returned error can't find the container with id ae6642774bd8b7cb9be5f78d2833985219e194cf07d50d7e0c8a1f417b539c9f Jul 18 02:39:37.858783 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:37.858757 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-17 02:34:37 +0000 UTC" deadline="2027-12-16 12:20:28.011275992 +0000 UTC" Jul 18 02:39:37.858851 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:37.858784 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12393h40m50.152496568s" Jul 18 02:39:37.859053 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:37.859032 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea3598a841ad3d314209c397f6def29f.slice/crio-ddc5d461f769d91c3e5b133be5ef32868ed57dba44aa4f5b6882636b138eb732 WatchSource:0}: Error finding container ddc5d461f769d91c3e5b133be5ef32868ed57dba44aa4f5b6882636b138eb732: Status 404 returned error can't find the container with id ddc5d461f769d91c3e5b133be5ef32868ed57dba44aa4f5b6882636b138eb732 Jul 18 02:39:37.861366 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:37.861346 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 02:39:37.874695 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:37.874679 2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 02:39:37.899137 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:37.899092 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-4.ec2.internal" event={"ID":"ea3598a841ad3d314209c397f6def29f","Type":"ContainerStarted","Data":"ddc5d461f769d91c3e5b133be5ef32868ed57dba44aa4f5b6882636b138eb732"} Jul 18 02:39:37.900122 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:37.900099 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-4.ec2.internal" event={"ID":"10427b699bfe492315ce6d6f3f8955b6","Type":"ContainerStarted","Data":"ae6642774bd8b7cb9be5f78d2833985219e194cf07d50d7e0c8a1f417b539c9f"} Jul 18 02:39:37.905693 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:37.905658 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-4.ec2.internal\" not found" Jul 18 02:39:37.930621 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:37.930602 2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 02:39:38.005646 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.005600 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-4.ec2.internal" Jul 18 02:39:38.020813 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.020791 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 18 02:39:38.022004 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.021990 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-4.ec2.internal" Jul 18 02:39:38.030287 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.030272 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 18 02:39:38.615284 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.615259 2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 02:39:38.781873 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.781844 2585 apiserver.go:52] "Watching apiserver" Jul 18 02:39:38.790364 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.790335 2585 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 18 02:39:38.791822 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.791796 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-8b2g5","openshift-network-diagnostics/network-check-target-8rfhc","openshift-cluster-node-tuning-operator/tuned-82mcs","openshift-dns/node-resolver-tlhsl","openshift-multus/multus-additional-cni-plugins-xvpps","openshift-multus/network-metrics-daemon-nht5x","openshift-network-operator/iptables-alerter-xt7md","openshift-ovn-kubernetes/ovnkube-node-kpl5z","kube-system/global-pull-secret-syncer-mqtcn","kube-system/konnectivity-agent-h6v9b","kube-system/kube-apiserver-proxy-ip-10-0-140-4.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6nsv","openshift-image-registry/node-ca-625jl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-4.ec2.internal"] Jul 18 02:39:38.794287 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.794260 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xt7md" Jul 18 02:39:38.797035 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.796977 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 18 02:39:38.797143 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.797041 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 18 02:39:38.797143 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.797050 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6qcc8\"" Jul 18 02:39:38.797375 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.797343 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 18 02:39:38.797638 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.797621 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-625jl" Jul 18 02:39:38.800664 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.800062 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.800664 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.800357 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 18 02:39:38.800664 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.800661 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pd8fw\"" Jul 18 02:39:38.800889 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.800662 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 18 02:39:38.800889 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.800755 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 18 02:39:38.802557 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.802457 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 18 02:39:38.803494 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.802881 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6ngvk\"" Jul 18 02:39:38.803494 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.803070 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 18 02:39:38.803494 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.803289 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tlhsl" Jul 18 02:39:38.805162 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.805140 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xvpps" Jul 18 02:39:38.805709 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.805660 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 18 02:39:38.805709 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.805684 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 18 02:39:38.805867 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.805775 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6hwpp\"" Jul 18 02:39:38.805867 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.805858 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 18 02:39:38.807690 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.807584 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 18 02:39:38.807690 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.807594 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:39:38.807690 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.807667 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 18 02:39:38.807996 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:38.807664 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nht5x" podUID="870f3a38-67e6-42ea-9398-f12eeb350f5d" Jul 18 02:39:38.808068 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.807998 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 18 02:39:38.808121 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.808096 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-c57cp\"" Jul 18 02:39:38.808175 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.808140 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 18 02:39:38.808221 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.808200 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 18 02:39:38.810247 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.810120 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:39:38.810247 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:38.810166 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqtcn" podUID="d8073815-9d72-4741-8447-ca9fa5dbd049" Jul 18 02:39:38.813474 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.813449 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8b2g5" Jul 18 02:39:38.815541 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.815523 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:39:38.815649 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:38.815590 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rfhc" podUID="01c7795c-7254-44d2-95eb-d3f076cd4bea" Jul 18 02:39:38.815888 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.815868 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 18 02:39:38.816101 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.816080 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-d9vnz\"" Jul 18 02:39:38.818328 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.818271 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h6v9b" Jul 18 02:39:38.818972 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.818775 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45jj8\" (UniqueName: \"kubernetes.io/projected/f24c6307-0556-4abf-ad93-fe954de837b4-kube-api-access-45jj8\") pod \"iptables-alerter-xt7md\" (UID: \"f24c6307-0556-4abf-ad93-fe954de837b4\") " pod="openshift-network-operator/iptables-alerter-xt7md" Jul 18 02:39:38.818972 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.818811 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1b59997b-ff58-43bc-b393-a6c42f6f1ed4-serviceca\") pod \"node-ca-625jl\" (UID: \"1b59997b-ff58-43bc-b393-a6c42f6f1ed4\") " pod="openshift-image-registry/node-ca-625jl" Jul 18 02:39:38.818972 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.818834 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-etc-sysconfig\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.818972 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.818856 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-etc-kubernetes\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.818972 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.818877 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6b049edb-9907-4ff4-af47-b54194c0776d-etc-tuned\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.818972 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.818908 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-etc-sysctl-d\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.819614 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.819585 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35cb46c9-0c8a-4562-a059-521ad01f2552-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xvpps\" (UID: \"35cb46c9-0c8a-4562-a059-521ad01f2552\") " pod="openshift-multus/multus-additional-cni-plugins-xvpps" Jul 18 02:39:38.819696 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.819647 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-etc-modprobe-d\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.819696 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.819680 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7321f9a8-9e03-4dff-a574-0175b0c89260-tmp-dir\") pod \"node-resolver-tlhsl\" (UID: \"7321f9a8-9e03-4dff-a574-0175b0c89260\") " pod="openshift-dns/node-resolver-tlhsl" Jul 18 02:39:38.819801 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.819712 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f24c6307-0556-4abf-ad93-fe954de837b4-iptables-alerter-script\") pod \"iptables-alerter-xt7md\" (UID: \"f24c6307-0556-4abf-ad93-fe954de837b4\") " pod="openshift-network-operator/iptables-alerter-xt7md" Jul 18 02:39:38.819801 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.819737 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-run\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.819801 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.819767 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7321f9a8-9e03-4dff-a574-0175b0c89260-hosts-file\") pod \"node-resolver-tlhsl\" (UID: \"7321f9a8-9e03-4dff-a574-0175b0c89260\") " pod="openshift-dns/node-resolver-tlhsl" Jul 18 02:39:38.819801 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.819798 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35cb46c9-0c8a-4562-a059-521ad01f2552-system-cni-dir\") pod \"multus-additional-cni-plugins-xvpps\" (UID: \"35cb46c9-0c8a-4562-a059-521ad01f2552\") " pod="openshift-multus/multus-additional-cni-plugins-xvpps" Jul 18 02:39:38.819982 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.819830 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-lib-modules\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.819982 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.819866 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-var-lib-kubelet\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.819982 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.819924 2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 18 02:39:38.820374 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.819892 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870f3a38-67e6-42ea-9398-f12eeb350f5d-metrics-certs\") pod \"network-metrics-daemon-nht5x\" (UID: \"870f3a38-67e6-42ea-9398-f12eeb350f5d\") " pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:39:38.820374 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.820337 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f24c6307-0556-4abf-ad93-fe954de837b4-host-slash\") pod \"iptables-alerter-xt7md\" (UID: \"f24c6307-0556-4abf-ad93-fe954de837b4\") " pod="openshift-network-operator/iptables-alerter-xt7md" Jul 18 02:39:38.820374 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.820366 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b59997b-ff58-43bc-b393-a6c42f6f1ed4-host\") pod \"node-ca-625jl\" (UID: \"1b59997b-ff58-43bc-b393-a6c42f6f1ed4\") " pod="openshift-image-registry/node-ca-625jl" Jul 18 02:39:38.822162 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.820403 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6mw6\" (UniqueName: \"kubernetes.io/projected/1b59997b-ff58-43bc-b393-a6c42f6f1ed4-kube-api-access-r6mw6\") pod \"node-ca-625jl\" (UID: \"1b59997b-ff58-43bc-b393-a6c42f6f1ed4\") " pod="openshift-image-registry/node-ca-625jl" Jul 18 02:39:38.822162 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.820447 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-etc-sysctl-conf\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.822162 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.820479 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-etc-systemd\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.822162 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.820509 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b049edb-9907-4ff4-af47-b54194c0776d-tmp\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.822162 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.820632 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35cb46c9-0c8a-4562-a059-521ad01f2552-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xvpps\" (UID: \"35cb46c9-0c8a-4562-a059-521ad01f2552\") " pod="openshift-multus/multus-additional-cni-plugins-xvpps" Jul 18 02:39:38.822162 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.820666 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35cb46c9-0c8a-4562-a059-521ad01f2552-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xvpps\" (UID: \"35cb46c9-0c8a-4562-a059-521ad01f2552\") " pod="openshift-multus/multus-additional-cni-plugins-xvpps" Jul 18 02:39:38.822162 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.820699 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-sys\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.822162 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.820738 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-host\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.822162 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.820773 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh4mz\" (UniqueName: \"kubernetes.io/projected/6b049edb-9907-4ff4-af47-b54194c0776d-kube-api-access-xh4mz\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.822162 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.820801 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35cb46c9-0c8a-4562-a059-521ad01f2552-cni-binary-copy\") pod \"multus-additional-cni-plugins-xvpps\" (UID: \"35cb46c9-0c8a-4562-a059-521ad01f2552\") " pod="openshift-multus/multus-additional-cni-plugins-xvpps" Jul 18 02:39:38.822162 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.820846 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzxcf\" (UniqueName: \"kubernetes.io/projected/35cb46c9-0c8a-4562-a059-521ad01f2552-kube-api-access-vzxcf\") pod \"multus-additional-cni-plugins-xvpps\" (UID: \"35cb46c9-0c8a-4562-a059-521ad01f2552\") " pod="openshift-multus/multus-additional-cni-plugins-xvpps" Jul 18 02:39:38.822162 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.820877 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgtvg\" (UniqueName: \"kubernetes.io/projected/7321f9a8-9e03-4dff-a574-0175b0c89260-kube-api-access-xgtvg\") pod \"node-resolver-tlhsl\" (UID: \"7321f9a8-9e03-4dff-a574-0175b0c89260\") " pod="openshift-dns/node-resolver-tlhsl" Jul 18 02:39:38.822162 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.820911 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35cb46c9-0c8a-4562-a059-521ad01f2552-cnibin\") pod \"multus-additional-cni-plugins-xvpps\" (UID: \"35cb46c9-0c8a-4562-a059-521ad01f2552\") " pod="openshift-multus/multus-additional-cni-plugins-xvpps" Jul 18 02:39:38.822162 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.821000 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/35cb46c9-0c8a-4562-a059-521ad01f2552-os-release\") pod \"multus-additional-cni-plugins-xvpps\" (UID: \"35cb46c9-0c8a-4562-a059-521ad01f2552\") " pod="openshift-multus/multus-additional-cni-plugins-xvpps" Jul 18 02:39:38.822162 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.821031 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgw4w\" (UniqueName: \"kubernetes.io/projected/870f3a38-67e6-42ea-9398-f12eeb350f5d-kube-api-access-xgw4w\") pod \"network-metrics-daemon-nht5x\" (UID: \"870f3a38-67e6-42ea-9398-f12eeb350f5d\") " pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:39:38.822162 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.821490 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 18 02:39:38.822162 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.821652 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8rn4k\"" Jul 18 02:39:38.822162 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.821757 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 18 02:39:38.823190 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.823169 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:38.825581 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.825560 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6nsv" Jul 18 02:39:38.825963 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.825940 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zf9mr\"" Jul 18 02:39:38.826177 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.826157 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 18 02:39:38.826285 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.826265 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 18 02:39:38.826443 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.826423 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 18 02:39:38.827170 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.827150 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 18 02:39:38.827296 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.827172 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 18 02:39:38.827296 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.827235 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 18 02:39:38.827927 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.827907 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 18 02:39:38.828343 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.828323 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 18 02:39:38.828723 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.828705 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 18 02:39:38.828805 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.828732 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-c9f2v\"" Jul 18 02:39:38.844381 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.844358 2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-q58sz" Jul 18 02:39:38.854160 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.854132 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-q58sz" Jul 18 02:39:38.907146 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.907127 2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 18 02:39:38.922313 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.922284 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f24c6307-0556-4abf-ad93-fe954de837b4-host-slash\") pod \"iptables-alerter-xt7md\" (UID: \"f24c6307-0556-4abf-ad93-fe954de837b4\") " pod="openshift-network-operator/iptables-alerter-xt7md" Jul 18 02:39:38.922313 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.922316 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-etc-sysctl-conf\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.922513 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.922337 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-host\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.922513 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.922371 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35cb46c9-0c8a-4562-a059-521ad01f2552-cni-binary-copy\") pod \"multus-additional-cni-plugins-xvpps\" (UID: \"35cb46c9-0c8a-4562-a059-521ad01f2552\") " pod="openshift-multus/multus-additional-cni-plugins-xvpps" Jul 18 02:39:38.922513 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.922392 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzxcf\" (UniqueName: \"kubernetes.io/projected/35cb46c9-0c8a-4562-a059-521ad01f2552-kube-api-access-vzxcf\") pod \"multus-additional-cni-plugins-xvpps\" (UID: \"35cb46c9-0c8a-4562-a059-521ad01f2552\") " pod="openshift-multus/multus-additional-cni-plugins-xvpps" Jul 18 02:39:38.922513 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.922413 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-var-lib-openvswitch\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:38.922513 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.922435 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdl8j\" (UniqueName: \"kubernetes.io/projected/01c7795c-7254-44d2-95eb-d3f076cd4bea-kube-api-access-fdl8j\") pod \"network-check-target-8rfhc\" (UID: \"01c7795c-7254-44d2-95eb-d3f076cd4bea\") " pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:39:38.922513 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.922466 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-os-release\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:38.922513 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.922485 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d56b8798-4bc3-4101-b769-c129c5097167-multus-daemon-config\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:38.922513 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.922507 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-etc-sysctl-conf\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.922894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.922368 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f24c6307-0556-4abf-ad93-fe954de837b4-host-slash\") pod \"iptables-alerter-xt7md\" (UID: \"f24c6307-0556-4abf-ad93-fe954de837b4\") " pod="openshift-network-operator/iptables-alerter-xt7md" Jul 18 02:39:38.922894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.922504 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-systemd-units\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:38.922894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.922570 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-node-log\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:38.922894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.922600 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6ee0c486-dd44-4685-9695-2dff5f42ace2-sys-fs\") pod \"aws-ebs-csi-driver-node-d6nsv\" (UID: \"6ee0c486-dd44-4685-9695-2dff5f42ace2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6nsv" Jul 18 02:39:38.922894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.922630 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1b59997b-ff58-43bc-b393-a6c42f6f1ed4-serviceca\") pod \"node-ca-625jl\" (UID: \"1b59997b-ff58-43bc-b393-a6c42f6f1ed4\") " pod="openshift-image-registry/node-ca-625jl" Jul 18 02:39:38.922894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.922655 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6b049edb-9907-4ff4-af47-b54194c0776d-etc-tuned\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.922894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.922682 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/76644748-24e4-4b1d-a6b1-68887436724c-konnectivity-ca\") pod \"konnectivity-agent-h6v9b\" (UID: \"76644748-24e4-4b1d-a6b1-68887436724c\") " pod="kube-system/konnectivity-agent-h6v9b" Jul 18 02:39:38.922894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.922707 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-host-var-lib-cni-multus\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:38.922894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.922730 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-host-var-lib-kubelet\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:38.922894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.922756 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-multus-conf-dir\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:38.922894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.922778 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-host-run-netns\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:38.922894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.922803 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:38.922894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.922831 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-host-run-multus-certs\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:38.922894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.922859 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7321f9a8-9e03-4dff-a574-0175b0c89260-tmp-dir\") pod \"node-resolver-tlhsl\" (UID: \"7321f9a8-9e03-4dff-a574-0175b0c89260\") " pod="openshift-dns/node-resolver-tlhsl" Jul 18 02:39:38.922894 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.922882 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-cnibin\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:38.923704 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.922923 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-host-run-k8s-cni-cncf-io\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:38.923704 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.922952 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-host-run-netns\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:38.923704 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.922966 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35cb46c9-0c8a-4562-a059-521ad01f2552-cni-binary-copy\") pod \"multus-additional-cni-plugins-xvpps\" (UID: \"35cb46c9-0c8a-4562-a059-521ad01f2552\") " pod="openshift-multus/multus-additional-cni-plugins-xvpps" Jul 18 02:39:38.923704 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.922999 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-host-var-lib-cni-bin\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:38.923704 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923031 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-host-cni-netd\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:38.923704 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923041 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-host\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.923704 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923026 2585 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 18 02:39:38.923704 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923060 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-run\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.923704 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923130 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7321f9a8-9e03-4dff-a574-0175b0c89260-hosts-file\") pod \"node-resolver-tlhsl\" (UID: \"7321f9a8-9e03-4dff-a574-0175b0c89260\") " pod="openshift-dns/node-resolver-tlhsl" Jul 18 02:39:38.923704 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923162 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35cb46c9-0c8a-4562-a059-521ad01f2552-system-cni-dir\") pod \"multus-additional-cni-plugins-xvpps\" (UID: \"35cb46c9-0c8a-4562-a059-521ad01f2552\") " pod="openshift-multus/multus-additional-cni-plugins-xvpps" Jul 18 02:39:38.923704 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923207 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-run\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.923704 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923207 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d8073815-9d72-4741-8447-ca9fa5dbd049-dbus\") pod \"global-pull-secret-syncer-mqtcn\" (UID: \"d8073815-9d72-4741-8447-ca9fa5dbd049\") " pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:39:38.923704 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923249 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35cb46c9-0c8a-4562-a059-521ad01f2552-system-cni-dir\") pod \"multus-additional-cni-plugins-xvpps\" (UID: \"35cb46c9-0c8a-4562-a059-521ad01f2552\") " pod="openshift-multus/multus-additional-cni-plugins-xvpps" Jul 18 02:39:38.923704 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923309 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-system-cni-dir\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:38.923704 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923341 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-multus-cni-dir\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:38.923704 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923356 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7321f9a8-9e03-4dff-a574-0175b0c89260-tmp-dir\") pod \"node-resolver-tlhsl\" (UID: \"7321f9a8-9e03-4dff-a574-0175b0c89260\") " pod="openshift-dns/node-resolver-tlhsl" Jul 18 02:39:38.923704 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923369 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nzrj\" (UniqueName: \"kubernetes.io/projected/d56b8798-4bc3-4101-b769-c129c5097167-kube-api-access-5nzrj\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:38.923704 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923393 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-run-systemd\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:38.924651 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923310 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7321f9a8-9e03-4dff-a574-0175b0c89260-hosts-file\") pod \"node-resolver-tlhsl\" (UID: \"7321f9a8-9e03-4dff-a574-0175b0c89260\") " pod="openshift-dns/node-resolver-tlhsl" Jul 18 02:39:38.924651 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923446 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870f3a38-67e6-42ea-9398-f12eeb350f5d-metrics-certs\") pod \"network-metrics-daemon-nht5x\" (UID: \"870f3a38-67e6-42ea-9398-f12eeb350f5d\") " pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:39:38.924651 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923468 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-etc-kubernetes\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:38.924651 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923510 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-run-ovn\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:38.924651 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923612 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-log-socket\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:38.924651 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923634 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1b59997b-ff58-43bc-b393-a6c42f6f1ed4-serviceca\") pod \"node-ca-625jl\" (UID: \"1b59997b-ff58-43bc-b393-a6c42f6f1ed4\") " pod="openshift-image-registry/node-ca-625jl" Jul 18 02:39:38.924651 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923662 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-host-cni-bin\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:38.924651 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923690 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwd2g\" (UniqueName: \"kubernetes.io/projected/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-kube-api-access-gwd2g\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:38.924651 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923713 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6ee0c486-dd44-4685-9695-2dff5f42ace2-device-dir\") pod \"aws-ebs-csi-driver-node-d6nsv\" (UID: \"6ee0c486-dd44-4685-9695-2dff5f42ace2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6nsv" Jul 18 02:39:38.924651 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923736 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b59997b-ff58-43bc-b393-a6c42f6f1ed4-host\") pod \"node-ca-625jl\" (UID: \"1b59997b-ff58-43bc-b393-a6c42f6f1ed4\") " pod="openshift-image-registry/node-ca-625jl" Jul 18 02:39:38.924651 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923757 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6mw6\" (UniqueName: \"kubernetes.io/projected/1b59997b-ff58-43bc-b393-a6c42f6f1ed4-kube-api-access-r6mw6\") pod \"node-ca-625jl\" (UID: \"1b59997b-ff58-43bc-b393-a6c42f6f1ed4\") " pod="openshift-image-registry/node-ca-625jl" Jul 18 02:39:38.924651 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923772 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-etc-systemd\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.924651 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923791 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b049edb-9907-4ff4-af47-b54194c0776d-tmp\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.924651 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923814 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35cb46c9-0c8a-4562-a059-521ad01f2552-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xvpps\" (UID: \"35cb46c9-0c8a-4562-a059-521ad01f2552\") " pod="openshift-multus/multus-additional-cni-plugins-xvpps" Jul 18 02:39:38.924651 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923840 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35cb46c9-0c8a-4562-a059-521ad01f2552-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xvpps\" (UID: \"35cb46c9-0c8a-4562-a059-521ad01f2552\") " pod="openshift-multus/multus-additional-cni-plugins-xvpps" Jul 18 02:39:38.924651 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923849 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b59997b-ff58-43bc-b393-a6c42f6f1ed4-host\") pod \"node-ca-625jl\" (UID: \"1b59997b-ff58-43bc-b393-a6c42f6f1ed4\") " pod="openshift-image-registry/node-ca-625jl" Jul 18 02:39:38.924651 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923861 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-etc-systemd\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.925539 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923866 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d8073815-9d72-4741-8447-ca9fa5dbd049-kubelet-config\") pod \"global-pull-secret-syncer-mqtcn\" (UID: \"d8073815-9d72-4741-8447-ca9fa5dbd049\") " pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:39:38.925539 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.923999 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d56b8798-4bc3-4101-b769-c129c5097167-cni-binary-copy\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:38.925539 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.924103 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-sys\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.925539 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:38.924107 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 02:39:38.925539 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.924165 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-sys\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.925539 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:38.924173 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/870f3a38-67e6-42ea-9398-f12eeb350f5d-metrics-certs podName:870f3a38-67e6-42ea-9398-f12eeb350f5d nodeName:}" failed. No retries permitted until 2026-07-18 02:39:39.424154702 +0000 UTC m=+3.027802390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/870f3a38-67e6-42ea-9398-f12eeb350f5d-metrics-certs") pod "network-metrics-daemon-nht5x" (UID: "870f3a38-67e6-42ea-9398-f12eeb350f5d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 02:39:38.925539 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.924250 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh4mz\" (UniqueName: \"kubernetes.io/projected/6b049edb-9907-4ff4-af47-b54194c0776d-kube-api-access-xh4mz\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.925539 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.924355 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35cb46c9-0c8a-4562-a059-521ad01f2552-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xvpps\" (UID: \"35cb46c9-0c8a-4562-a059-521ad01f2552\") " pod="openshift-multus/multus-additional-cni-plugins-xvpps" Jul 18 02:39:38.925539 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.924382 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-env-overrides\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:38.925539 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.924410 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6ee0c486-dd44-4685-9695-2dff5f42ace2-etc-selinux\") pod \"aws-ebs-csi-driver-node-d6nsv\" (UID: \"6ee0c486-dd44-4685-9695-2dff5f42ace2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6nsv" Jul 18 02:39:38.925539 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.924438 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgtvg\" (UniqueName: \"kubernetes.io/projected/7321f9a8-9e03-4dff-a574-0175b0c89260-kube-api-access-xgtvg\") pod \"node-resolver-tlhsl\" (UID: \"7321f9a8-9e03-4dff-a574-0175b0c89260\") " pod="openshift-dns/node-resolver-tlhsl" Jul 18 02:39:38.925539 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.924437 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35cb46c9-0c8a-4562-a059-521ad01f2552-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xvpps\" (UID: \"35cb46c9-0c8a-4562-a059-521ad01f2552\") " pod="openshift-multus/multus-additional-cni-plugins-xvpps" Jul 18 02:39:38.925539 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.924468 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35cb46c9-0c8a-4562-a059-521ad01f2552-cnibin\") pod \"multus-additional-cni-plugins-xvpps\" (UID: \"35cb46c9-0c8a-4562-a059-521ad01f2552\") " pod="openshift-multus/multus-additional-cni-plugins-xvpps" Jul 18 02:39:38.925539 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.924585 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35cb46c9-0c8a-4562-a059-521ad01f2552-cnibin\") pod \"multus-additional-cni-plugins-xvpps\" (UID: \"35cb46c9-0c8a-4562-a059-521ad01f2552\") " pod="openshift-multus/multus-additional-cni-plugins-xvpps" Jul 18 02:39:38.925539 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.924604 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/35cb46c9-0c8a-4562-a059-521ad01f2552-os-release\") pod \"multus-additional-cni-plugins-xvpps\" (UID: \"35cb46c9-0c8a-4562-a059-521ad01f2552\") " pod="openshift-multus/multus-additional-cni-plugins-xvpps" Jul 18 02:39:38.925539 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.924629 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgw4w\" (UniqueName: \"kubernetes.io/projected/870f3a38-67e6-42ea-9398-f12eeb350f5d-kube-api-access-xgw4w\") pod \"network-metrics-daemon-nht5x\" (UID: \"870f3a38-67e6-42ea-9398-f12eeb350f5d\") " pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:39:38.926371 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.924647 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-ovnkube-script-lib\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:38.926371 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.924665 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6ee0c486-dd44-4685-9695-2dff5f42ace2-socket-dir\") pod \"aws-ebs-csi-driver-node-d6nsv\" (UID: \"6ee0c486-dd44-4685-9695-2dff5f42ace2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6nsv" Jul 18 02:39:38.926371 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.924687 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smm6x\" (UniqueName: \"kubernetes.io/projected/6ee0c486-dd44-4685-9695-2dff5f42ace2-kube-api-access-smm6x\") pod \"aws-ebs-csi-driver-node-d6nsv\" (UID: \"6ee0c486-dd44-4685-9695-2dff5f42ace2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6nsv" Jul 18 02:39:38.926371 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.924699 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/35cb46c9-0c8a-4562-a059-521ad01f2552-os-release\") pod \"multus-additional-cni-plugins-xvpps\" (UID: \"35cb46c9-0c8a-4562-a059-521ad01f2552\") " pod="openshift-multus/multus-additional-cni-plugins-xvpps" Jul 18 02:39:38.926371 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.924708 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45jj8\" (UniqueName: \"kubernetes.io/projected/f24c6307-0556-4abf-ad93-fe954de837b4-kube-api-access-45jj8\") pod \"iptables-alerter-xt7md\" (UID: \"f24c6307-0556-4abf-ad93-fe954de837b4\") " pod="openshift-network-operator/iptables-alerter-xt7md" Jul 18 02:39:38.926371 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.924724 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-etc-sysconfig\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.926371 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.924751 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-etc-kubernetes\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.926371 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.924799 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-etc-kubernetes\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.926371 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.924830 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-host-kubelet\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:38.926371 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.924857 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-etc-sysctl-d\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.926371 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.924954 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-etc-sysctl-d\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.926371 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.924961 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35cb46c9-0c8a-4562-a059-521ad01f2552-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xvpps\" (UID: \"35cb46c9-0c8a-4562-a059-521ad01f2552\") " pod="openshift-multus/multus-additional-cni-plugins-xvpps" Jul 18 02:39:38.926371 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.925008 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-etc-openvswitch\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:38.926371 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.925051 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-etc-modprobe-d\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.926371 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.925072 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35cb46c9-0c8a-4562-a059-521ad01f2552-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xvpps\" (UID: \"35cb46c9-0c8a-4562-a059-521ad01f2552\") " pod="openshift-multus/multus-additional-cni-plugins-xvpps" Jul 18 02:39:38.926371 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.925092 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8073815-9d72-4741-8447-ca9fa5dbd049-original-pull-secret\") pod \"global-pull-secret-syncer-mqtcn\" (UID: \"d8073815-9d72-4741-8447-ca9fa5dbd049\") " pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:39:38.926371 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.925114 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-etc-sysconfig\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.927326 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.925121 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/76644748-24e4-4b1d-a6b1-68887436724c-agent-certs\") pod \"konnectivity-agent-h6v9b\" (UID: \"76644748-24e4-4b1d-a6b1-68887436724c\") " pod="kube-system/konnectivity-agent-h6v9b" Jul 18 02:39:38.927326 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.925153 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-host-slash\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:38.927326 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.925184 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-ovnkube-config\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:38.927326 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.925210 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-etc-modprobe-d\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.927326 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.925209 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-ovn-node-metrics-cert\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:38.927326 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.925245 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6ee0c486-dd44-4685-9695-2dff5f42ace2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d6nsv\" (UID: \"6ee0c486-dd44-4685-9695-2dff5f42ace2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6nsv" Jul 18 02:39:38.927326 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.925269 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f24c6307-0556-4abf-ad93-fe954de837b4-iptables-alerter-script\") pod \"iptables-alerter-xt7md\" (UID: \"f24c6307-0556-4abf-ad93-fe954de837b4\") " pod="openshift-network-operator/iptables-alerter-xt7md" Jul 18 02:39:38.927326 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.925293 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-host-run-ovn-kubernetes\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:38.927326 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.925311 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6ee0c486-dd44-4685-9695-2dff5f42ace2-registration-dir\") pod \"aws-ebs-csi-driver-node-d6nsv\" (UID: \"6ee0c486-dd44-4685-9695-2dff5f42ace2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6nsv" Jul 18 02:39:38.927326 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.925334 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-lib-modules\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.927326 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.925402 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-var-lib-kubelet\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.927326 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.925418 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-lib-modules\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.927326 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.925437 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-multus-socket-dir-parent\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:38.927326 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.925458 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b049edb-9907-4ff4-af47-b54194c0776d-var-lib-kubelet\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.927326 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.925462 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-hostroot\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:38.927326 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.925488 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-run-openvswitch\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:38.927326 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.925844 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f24c6307-0556-4abf-ad93-fe954de837b4-iptables-alerter-script\") pod \"iptables-alerter-xt7md\" (UID: \"f24c6307-0556-4abf-ad93-fe954de837b4\") " pod="openshift-network-operator/iptables-alerter-xt7md" Jul 18 02:39:38.928187 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.926695 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b049edb-9907-4ff4-af47-b54194c0776d-tmp\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.928187 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.926759 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6b049edb-9907-4ff4-af47-b54194c0776d-etc-tuned\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.934552 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.934523 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgw4w\" (UniqueName: \"kubernetes.io/projected/870f3a38-67e6-42ea-9398-f12eeb350f5d-kube-api-access-xgw4w\") pod \"network-metrics-daemon-nht5x\" (UID: \"870f3a38-67e6-42ea-9398-f12eeb350f5d\") " pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:39:38.934552 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.934536 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45jj8\" (UniqueName: \"kubernetes.io/projected/f24c6307-0556-4abf-ad93-fe954de837b4-kube-api-access-45jj8\") pod \"iptables-alerter-xt7md\" (UID: \"f24c6307-0556-4abf-ad93-fe954de837b4\") " pod="openshift-network-operator/iptables-alerter-xt7md" Jul 18 02:39:38.935002 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.934974 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6mw6\" (UniqueName: \"kubernetes.io/projected/1b59997b-ff58-43bc-b393-a6c42f6f1ed4-kube-api-access-r6mw6\") pod \"node-ca-625jl\" (UID: \"1b59997b-ff58-43bc-b393-a6c42f6f1ed4\") " pod="openshift-image-registry/node-ca-625jl" Jul 18 02:39:38.935105 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.935002 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzxcf\" (UniqueName: \"kubernetes.io/projected/35cb46c9-0c8a-4562-a059-521ad01f2552-kube-api-access-vzxcf\") pod \"multus-additional-cni-plugins-xvpps\" (UID: \"35cb46c9-0c8a-4562-a059-521ad01f2552\") " pod="openshift-multus/multus-additional-cni-plugins-xvpps" Jul 18 02:39:38.935105 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.935002 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh4mz\" (UniqueName: \"kubernetes.io/projected/6b049edb-9907-4ff4-af47-b54194c0776d-kube-api-access-xh4mz\") pod \"tuned-82mcs\" (UID: \"6b049edb-9907-4ff4-af47-b54194c0776d\") " pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:38.935608 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:38.935586 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgtvg\" (UniqueName: \"kubernetes.io/projected/7321f9a8-9e03-4dff-a574-0175b0c89260-kube-api-access-xgtvg\") pod \"node-resolver-tlhsl\" (UID: \"7321f9a8-9e03-4dff-a574-0175b0c89260\") " pod="openshift-dns/node-resolver-tlhsl" Jul 18 02:39:39.025750 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.025715 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-var-lib-openvswitch\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.025928 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.025758 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdl8j\" (UniqueName: \"kubernetes.io/projected/01c7795c-7254-44d2-95eb-d3f076cd4bea-kube-api-access-fdl8j\") pod \"network-check-target-8rfhc\" (UID: \"01c7795c-7254-44d2-95eb-d3f076cd4bea\") " pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:39:39.025928 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.025782 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-os-release\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.025928 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.025807 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d56b8798-4bc3-4101-b769-c129c5097167-multus-daemon-config\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.025928 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.025829 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-systemd-units\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.025928 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.025834 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-var-lib-openvswitch\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.025928 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.025850 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-node-log\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.025928 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.025869 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-os-release\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.025928 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.025891 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-node-log\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.025928 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.025890 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6ee0c486-dd44-4685-9695-2dff5f42ace2-sys-fs\") pod \"aws-ebs-csi-driver-node-d6nsv\" (UID: \"6ee0c486-dd44-4685-9695-2dff5f42ace2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6nsv" Jul 18 02:39:39.025928 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.025925 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/76644748-24e4-4b1d-a6b1-68887436724c-konnectivity-ca\") pod \"konnectivity-agent-h6v9b\" (UID: \"76644748-24e4-4b1d-a6b1-68887436724c\") " pod="kube-system/konnectivity-agent-h6v9b" Jul 18 02:39:39.025928 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.025932 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6ee0c486-dd44-4685-9695-2dff5f42ace2-sys-fs\") pod \"aws-ebs-csi-driver-node-d6nsv\" (UID: \"6ee0c486-dd44-4685-9695-2dff5f42ace2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6nsv" Jul 18 02:39:39.026514 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.025951 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-host-var-lib-cni-multus\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.026514 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.025975 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-host-var-lib-kubelet\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.026514 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.025989 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-systemd-units\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.026514 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026011 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-multus-conf-dir\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.026514 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026037 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-host-run-netns\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.026514 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026062 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-host-var-lib-cni-multus\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.026514 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026079 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.026514 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026113 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-multus-conf-dir\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.026514 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026108 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-host-run-multus-certs\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.026514 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026153 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-cnibin\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.026514 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026179 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-host-run-k8s-cni-cncf-io\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.026514 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026199 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.026514 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026204 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-host-run-netns\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.026514 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026249 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-host-var-lib-cni-bin\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.026514 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026259 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-host-var-lib-kubelet\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.026514 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026275 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-host-cni-netd\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.026514 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026302 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d8073815-9d72-4741-8447-ca9fa5dbd049-dbus\") pod \"global-pull-secret-syncer-mqtcn\" (UID: \"d8073815-9d72-4741-8447-ca9fa5dbd049\") " pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:39:39.026514 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026312 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-cnibin\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.027455 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026155 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-host-run-multus-certs\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.027455 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026327 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-system-cni-dir\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.027455 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026367 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-host-run-netns\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.027455 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026369 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-multus-cni-dir\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.027455 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026410 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nzrj\" (UniqueName: \"kubernetes.io/projected/d56b8798-4bc3-4101-b769-c129c5097167-kube-api-access-5nzrj\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.027455 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026431 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-run-systemd\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.027455 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026465 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-etc-kubernetes\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.027455 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026487 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d8073815-9d72-4741-8447-ca9fa5dbd049-dbus\") pod \"global-pull-secret-syncer-mqtcn\" (UID: \"d8073815-9d72-4741-8447-ca9fa5dbd049\") " pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:39:39.027455 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026497 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-run-ovn\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.027455 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026519 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/76644748-24e4-4b1d-a6b1-68887436724c-konnectivity-ca\") pod \"konnectivity-agent-h6v9b\" (UID: \"76644748-24e4-4b1d-a6b1-68887436724c\") " pod="kube-system/konnectivity-agent-h6v9b" Jul 18 02:39:39.027455 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026522 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-log-socket\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.027455 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026542 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-system-cni-dir\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.027455 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026553 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-host-run-k8s-cni-cncf-io\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.027455 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026556 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-host-cni-bin\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.027455 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026519 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d56b8798-4bc3-4101-b769-c129c5097167-multus-daemon-config\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.027455 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026576 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-run-systemd\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.027455 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026581 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-multus-cni-dir\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.027455 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026588 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-etc-kubernetes\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.028205 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026594 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwd2g\" (UniqueName: \"kubernetes.io/projected/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-kube-api-access-gwd2g\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.028205 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026621 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6ee0c486-dd44-4685-9695-2dff5f42ace2-device-dir\") pod \"aws-ebs-csi-driver-node-d6nsv\" (UID: \"6ee0c486-dd44-4685-9695-2dff5f42ace2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6nsv" Jul 18 02:39:39.028205 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026628 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-host-var-lib-cni-bin\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.028205 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026629 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-run-ovn\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.028205 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026641 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-log-socket\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.028205 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026653 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d8073815-9d72-4741-8447-ca9fa5dbd049-kubelet-config\") pod \"global-pull-secret-syncer-mqtcn\" (UID: \"d8073815-9d72-4741-8447-ca9fa5dbd049\") " pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:39:39.028205 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026671 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-host-cni-bin\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.028205 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026679 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d56b8798-4bc3-4101-b769-c129c5097167-cni-binary-copy\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.028205 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026686 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6ee0c486-dd44-4685-9695-2dff5f42ace2-device-dir\") pod \"aws-ebs-csi-driver-node-d6nsv\" (UID: \"6ee0c486-dd44-4685-9695-2dff5f42ace2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6nsv" Jul 18 02:39:39.028205 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026524 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-host-run-netns\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.028205 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026695 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d8073815-9d72-4741-8447-ca9fa5dbd049-kubelet-config\") pod \"global-pull-secret-syncer-mqtcn\" (UID: \"d8073815-9d72-4741-8447-ca9fa5dbd049\") " pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:39:39.028205 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026656 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-host-cni-netd\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.028205 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026711 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-env-overrides\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.028205 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026750 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6ee0c486-dd44-4685-9695-2dff5f42ace2-etc-selinux\") pod \"aws-ebs-csi-driver-node-d6nsv\" (UID: \"6ee0c486-dd44-4685-9695-2dff5f42ace2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6nsv" Jul 18 02:39:39.028205 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026781 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-ovnkube-script-lib\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.028205 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026808 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6ee0c486-dd44-4685-9695-2dff5f42ace2-socket-dir\") pod \"aws-ebs-csi-driver-node-d6nsv\" (UID: \"6ee0c486-dd44-4685-9695-2dff5f42ace2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6nsv" Jul 18 02:39:39.028205 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026835 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smm6x\" (UniqueName: \"kubernetes.io/projected/6ee0c486-dd44-4685-9695-2dff5f42ace2-kube-api-access-smm6x\") pod \"aws-ebs-csi-driver-node-d6nsv\" (UID: \"6ee0c486-dd44-4685-9695-2dff5f42ace2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6nsv" Jul 18 02:39:39.028808 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026874 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-host-kubelet\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.028808 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026882 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6ee0c486-dd44-4685-9695-2dff5f42ace2-etc-selinux\") pod \"aws-ebs-csi-driver-node-d6nsv\" (UID: \"6ee0c486-dd44-4685-9695-2dff5f42ace2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6nsv" Jul 18 02:39:39.028808 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026923 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-host-kubelet\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.028808 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026951 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-etc-openvswitch\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.028808 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.026980 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8073815-9d72-4741-8447-ca9fa5dbd049-original-pull-secret\") pod \"global-pull-secret-syncer-mqtcn\" (UID: \"d8073815-9d72-4741-8447-ca9fa5dbd049\") " pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:39:39.028808 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.027003 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/76644748-24e4-4b1d-a6b1-68887436724c-agent-certs\") pod \"konnectivity-agent-h6v9b\" (UID: \"76644748-24e4-4b1d-a6b1-68887436724c\") " pod="kube-system/konnectivity-agent-h6v9b" Jul 18 02:39:39.028808 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.027029 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-host-slash\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.028808 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.027054 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-ovnkube-config\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.028808 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.027078 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-ovn-node-metrics-cert\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.028808 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.027103 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6ee0c486-dd44-4685-9695-2dff5f42ace2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d6nsv\" (UID: \"6ee0c486-dd44-4685-9695-2dff5f42ace2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6nsv" Jul 18 02:39:39.028808 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.027130 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-host-run-ovn-kubernetes\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.028808 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.027134 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-env-overrides\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.028808 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.027156 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6ee0c486-dd44-4685-9695-2dff5f42ace2-registration-dir\") pod \"aws-ebs-csi-driver-node-d6nsv\" (UID: \"6ee0c486-dd44-4685-9695-2dff5f42ace2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6nsv" Jul 18 02:39:39.028808 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.027168 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-host-slash\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.028808 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.027185 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-multus-socket-dir-parent\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.028808 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.027204 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d56b8798-4bc3-4101-b769-c129c5097167-cni-binary-copy\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.028808 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.027210 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-hostroot\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.028808 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:39.027222 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 02:39:39.029416 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.027235 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-run-openvswitch\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.029416 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.027254 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-etc-openvswitch\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.029416 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:39.027293 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8073815-9d72-4741-8447-ca9fa5dbd049-original-pull-secret podName:d8073815-9d72-4741-8447-ca9fa5dbd049 nodeName:}" failed. No retries permitted until 2026-07-18 02:39:39.527257136 +0000 UTC m=+3.130904808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8073815-9d72-4741-8447-ca9fa5dbd049-original-pull-secret") pod "global-pull-secret-syncer-mqtcn" (UID: "d8073815-9d72-4741-8447-ca9fa5dbd049") : object "kube-system"/"original-pull-secret" not registered Jul 18 02:39:39.029416 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.027340 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-run-openvswitch\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.029416 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.027388 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-host-run-ovn-kubernetes\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.029416 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.027456 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-ovnkube-script-lib\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.029416 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.027518 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6ee0c486-dd44-4685-9695-2dff5f42ace2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d6nsv\" (UID: \"6ee0c486-dd44-4685-9695-2dff5f42ace2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6nsv" Jul 18 02:39:39.029416 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.027523 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6ee0c486-dd44-4685-9695-2dff5f42ace2-registration-dir\") pod \"aws-ebs-csi-driver-node-d6nsv\" (UID: \"6ee0c486-dd44-4685-9695-2dff5f42ace2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6nsv" Jul 18 02:39:39.029416 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.027130 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6ee0c486-dd44-4685-9695-2dff5f42ace2-socket-dir\") pod \"aws-ebs-csi-driver-node-d6nsv\" (UID: \"6ee0c486-dd44-4685-9695-2dff5f42ace2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6nsv" Jul 18 02:39:39.029416 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.027573 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-multus-socket-dir-parent\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.029416 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.027599 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d56b8798-4bc3-4101-b769-c129c5097167-hostroot\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.029416 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.027811 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-ovnkube-config\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.030018 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.029784 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-ovn-node-metrics-cert\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.030085 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.030064 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/76644748-24e4-4b1d-a6b1-68887436724c-agent-certs\") pod \"konnectivity-agent-h6v9b\" (UID: \"76644748-24e4-4b1d-a6b1-68887436724c\") " pod="kube-system/konnectivity-agent-h6v9b" Jul 18 02:39:39.035726 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:39.035707 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 02:39:39.035835 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:39.035732 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 02:39:39.035835 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:39.035742 2585 projected.go:196] Error preparing data for projected volume kube-api-access-fdl8j for pod openshift-network-diagnostics/network-check-target-8rfhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 02:39:39.035835 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:39.035802 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01c7795c-7254-44d2-95eb-d3f076cd4bea-kube-api-access-fdl8j podName:01c7795c-7254-44d2-95eb-d3f076cd4bea nodeName:}" failed. No retries permitted until 2026-07-18 02:39:39.535786098 +0000 UTC m=+3.139433766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fdl8j" (UniqueName: "kubernetes.io/projected/01c7795c-7254-44d2-95eb-d3f076cd4bea-kube-api-access-fdl8j") pod "network-check-target-8rfhc" (UID: "01c7795c-7254-44d2-95eb-d3f076cd4bea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 02:39:39.042351 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.042304 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smm6x\" (UniqueName: \"kubernetes.io/projected/6ee0c486-dd44-4685-9695-2dff5f42ace2-kube-api-access-smm6x\") pod \"aws-ebs-csi-driver-node-d6nsv\" (UID: \"6ee0c486-dd44-4685-9695-2dff5f42ace2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6nsv" Jul 18 02:39:39.042614 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.042590 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwd2g\" (UniqueName: \"kubernetes.io/projected/6b57a8b2-d5b6-4cce-9fb0-a3746be0de18-kube-api-access-gwd2g\") pod \"ovnkube-node-kpl5z\" (UID: \"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.042817 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.042790 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nzrj\" (UniqueName: \"kubernetes.io/projected/d56b8798-4bc3-4101-b769-c129c5097167-kube-api-access-5nzrj\") pod \"multus-8b2g5\" (UID: \"d56b8798-4bc3-4101-b769-c129c5097167\") " pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.109061 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.109028 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xt7md" Jul 18 02:39:39.117273 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.117254 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-625jl" Jul 18 02:39:39.129336 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.129287 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-82mcs" Jul 18 02:39:39.136234 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.136212 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tlhsl" Jul 18 02:39:39.145991 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.145968 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xvpps" Jul 18 02:39:39.152800 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.152771 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8b2g5" Jul 18 02:39:39.159527 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.159506 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h6v9b" Jul 18 02:39:39.168411 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.168392 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:39:39.173305 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.173287 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6nsv" Jul 18 02:39:39.430657 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.430576 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870f3a38-67e6-42ea-9398-f12eeb350f5d-metrics-certs\") pod \"network-metrics-daemon-nht5x\" (UID: \"870f3a38-67e6-42ea-9398-f12eeb350f5d\") " pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:39:39.430807 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:39.430731 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 02:39:39.430869 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:39.430808 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/870f3a38-67e6-42ea-9398-f12eeb350f5d-metrics-certs podName:870f3a38-67e6-42ea-9398-f12eeb350f5d nodeName:}" failed. No retries permitted until 2026-07-18 02:39:40.430787675 +0000 UTC m=+4.034435345 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/870f3a38-67e6-42ea-9398-f12eeb350f5d-metrics-certs") pod "network-metrics-daemon-nht5x" (UID: "870f3a38-67e6-42ea-9398-f12eeb350f5d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 02:39:39.531663 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.531615 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8073815-9d72-4741-8447-ca9fa5dbd049-original-pull-secret\") pod \"global-pull-secret-syncer-mqtcn\" (UID: \"d8073815-9d72-4741-8447-ca9fa5dbd049\") " pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:39:39.531842 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:39.531750 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 02:39:39.531842 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:39.531814 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8073815-9d72-4741-8447-ca9fa5dbd049-original-pull-secret podName:d8073815-9d72-4741-8447-ca9fa5dbd049 nodeName:}" failed. No retries permitted until 2026-07-18 02:39:40.531795658 +0000 UTC m=+4.135443329 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8073815-9d72-4741-8447-ca9fa5dbd049-original-pull-secret") pod "global-pull-secret-syncer-mqtcn" (UID: "d8073815-9d72-4741-8447-ca9fa5dbd049") : object "kube-system"/"original-pull-secret" not registered Jul 18 02:39:39.632539 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.632501 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdl8j\" (UniqueName: \"kubernetes.io/projected/01c7795c-7254-44d2-95eb-d3f076cd4bea-kube-api-access-fdl8j\") pod \"network-check-target-8rfhc\" (UID: \"01c7795c-7254-44d2-95eb-d3f076cd4bea\") " pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:39:39.632712 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:39.632634 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 02:39:39.632712 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:39.632653 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 02:39:39.632712 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:39.632667 2585 projected.go:196] Error preparing data for projected volume kube-api-access-fdl8j for pod openshift-network-diagnostics/network-check-target-8rfhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 02:39:39.632888 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:39.632736 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01c7795c-7254-44d2-95eb-d3f076cd4bea-kube-api-access-fdl8j podName:01c7795c-7254-44d2-95eb-d3f076cd4bea nodeName:}" failed. No retries permitted until 2026-07-18 02:39:40.632715499 +0000 UTC m=+4.236363186 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fdl8j" (UniqueName: "kubernetes.io/projected/01c7795c-7254-44d2-95eb-d3f076cd4bea-kube-api-access-fdl8j") pod "network-check-target-8rfhc" (UID: "01c7795c-7254-44d2-95eb-d3f076cd4bea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 02:39:39.858293 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.858259 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-17 02:34:39 +0000 UTC" deadline="2028-03-07 03:28:15.335888771 +0000 UTC" Jul 18 02:39:39.858293 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:39.858288 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14352h48m35.477605399s" Jul 18 02:39:40.005982 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:40.005950 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b59997b_ff58_43bc_b393_a6c42f6f1ed4.slice/crio-881e0f265fb4f30dd25672960088a05ac79304c8418e85c38a34b8c088ae7518 WatchSource:0}: Error finding container 881e0f265fb4f30dd25672960088a05ac79304c8418e85c38a34b8c088ae7518: Status 404 returned error can't find the container with id 881e0f265fb4f30dd25672960088a05ac79304c8418e85c38a34b8c088ae7518 Jul 18 02:39:40.007013 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:40.006987 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35cb46c9_0c8a_4562_a059_521ad01f2552.slice/crio-59f1532ddb5057c7cd78e8a6e1820a3a03b18ad46fcdaa86a02b7a59f4fcca74 WatchSource:0}: Error finding container 59f1532ddb5057c7cd78e8a6e1820a3a03b18ad46fcdaa86a02b7a59f4fcca74: Status 404 returned error can't find the container with id 59f1532ddb5057c7cd78e8a6e1820a3a03b18ad46fcdaa86a02b7a59f4fcca74 Jul 18 02:39:40.007569 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:40.007533 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76644748_24e4_4b1d_a6b1_68887436724c.slice/crio-48c8f3b94786775e2c5b2cf56b2629d28e528ed9f37ec462ea66455f880530ea WatchSource:0}: Error finding container 48c8f3b94786775e2c5b2cf56b2629d28e528ed9f37ec462ea66455f880530ea: Status 404 returned error can't find the container with id 48c8f3b94786775e2c5b2cf56b2629d28e528ed9f37ec462ea66455f880530ea Jul 18 02:39:40.009638 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:40.009151 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7321f9a8_9e03_4dff_a574_0175b0c89260.slice/crio-47dba9197728d8d202f1f151c038f9b13d03069363d1e75e318dd3f65180c339 WatchSource:0}: Error finding container 47dba9197728d8d202f1f151c038f9b13d03069363d1e75e318dd3f65180c339: Status 404 returned error can't find the container with id 47dba9197728d8d202f1f151c038f9b13d03069363d1e75e318dd3f65180c339 Jul 18 02:39:40.012391 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:40.012073 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee0c486_dd44_4685_9695_2dff5f42ace2.slice/crio-277a0cfa2f03e63443141e35a4a7ff2095c331e22e15086439ce7c374ea520b1 WatchSource:0}: Error finding container 277a0cfa2f03e63443141e35a4a7ff2095c331e22e15086439ce7c374ea520b1: Status 404 returned error can't find the container with id 277a0cfa2f03e63443141e35a4a7ff2095c331e22e15086439ce7c374ea520b1 Jul 18 02:39:40.012391 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:40.012375 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b57a8b2_d5b6_4cce_9fb0_a3746be0de18.slice/crio-1ed27ae7f9b232ced0f5e6c4792996fb5d87d19c9ec2fb901b96c3465666ab4b WatchSource:0}: Error finding container 1ed27ae7f9b232ced0f5e6c4792996fb5d87d19c9ec2fb901b96c3465666ab4b: Status 404 returned error can't find the container with id 1ed27ae7f9b232ced0f5e6c4792996fb5d87d19c9ec2fb901b96c3465666ab4b Jul 18 02:39:40.014007 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:40.013582 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56b8798_4bc3_4101_b769_c129c5097167.slice/crio-4679e0f7e10b5c6750944a5666b56828f8ca7d261dc082b29466ac003f107e4f WatchSource:0}: Error finding container 4679e0f7e10b5c6750944a5666b56828f8ca7d261dc082b29466ac003f107e4f: Status 404 returned error can't find the container with id 4679e0f7e10b5c6750944a5666b56828f8ca7d261dc082b29466ac003f107e4f Jul 18 02:39:40.015005 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:40.014814 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf24c6307_0556_4abf_ad93_fe954de837b4.slice/crio-6b0eba848ff61879da1e430e8da772d3b092dfb6fe981ab947d4d6b26a3746cd WatchSource:0}: Error finding container 6b0eba848ff61879da1e430e8da772d3b092dfb6fe981ab947d4d6b26a3746cd: Status 404 returned error can't find the container with id 6b0eba848ff61879da1e430e8da772d3b092dfb6fe981ab947d4d6b26a3746cd Jul 18 02:39:40.015752 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:39:40.015535 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b049edb_9907_4ff4_af47_b54194c0776d.slice/crio-48cf66a048888f4c410f73fe459bd20ac4b9820c1861245c70aeb06b1eb86cf1 WatchSource:0}: Error finding container 48cf66a048888f4c410f73fe459bd20ac4b9820c1861245c70aeb06b1eb86cf1: Status 404 returned error can't find the container with id 48cf66a048888f4c410f73fe459bd20ac4b9820c1861245c70aeb06b1eb86cf1 Jul 18 02:39:40.440870 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:40.440836 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870f3a38-67e6-42ea-9398-f12eeb350f5d-metrics-certs\") pod \"network-metrics-daemon-nht5x\" (UID: \"870f3a38-67e6-42ea-9398-f12eeb350f5d\") " pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:39:40.440992 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:40.440955 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 02:39:40.441041 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:40.441021 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/870f3a38-67e6-42ea-9398-f12eeb350f5d-metrics-certs podName:870f3a38-67e6-42ea-9398-f12eeb350f5d nodeName:}" failed. No retries permitted until 2026-07-18 02:39:42.441004379 +0000 UTC m=+6.044652045 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/870f3a38-67e6-42ea-9398-f12eeb350f5d-metrics-certs") pod "network-metrics-daemon-nht5x" (UID: "870f3a38-67e6-42ea-9398-f12eeb350f5d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 02:39:40.455197 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:40.455173 2585 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 02:39:40.541804 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:40.541777 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8073815-9d72-4741-8447-ca9fa5dbd049-original-pull-secret\") pod \"global-pull-secret-syncer-mqtcn\" (UID: \"d8073815-9d72-4741-8447-ca9fa5dbd049\") " pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:39:40.541924 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:40.541873 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 02:39:40.541924 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:40.541919 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8073815-9d72-4741-8447-ca9fa5dbd049-original-pull-secret podName:d8073815-9d72-4741-8447-ca9fa5dbd049 nodeName:}" failed. No retries permitted until 2026-07-18 02:39:42.541906838 +0000 UTC m=+6.145554505 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8073815-9d72-4741-8447-ca9fa5dbd049-original-pull-secret") pod "global-pull-secret-syncer-mqtcn" (UID: "d8073815-9d72-4741-8447-ca9fa5dbd049") : object "kube-system"/"original-pull-secret" not registered Jul 18 02:39:40.642593 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:40.642549 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdl8j\" (UniqueName: \"kubernetes.io/projected/01c7795c-7254-44d2-95eb-d3f076cd4bea-kube-api-access-fdl8j\") pod \"network-check-target-8rfhc\" (UID: \"01c7795c-7254-44d2-95eb-d3f076cd4bea\") " pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:39:40.642790 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:40.642737 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 02:39:40.642790 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:40.642757 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 02:39:40.642790 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:40.642771 2585 projected.go:196] Error preparing data for projected volume kube-api-access-fdl8j for pod openshift-network-diagnostics/network-check-target-8rfhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 02:39:40.643743 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:40.643701 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01c7795c-7254-44d2-95eb-d3f076cd4bea-kube-api-access-fdl8j podName:01c7795c-7254-44d2-95eb-d3f076cd4bea nodeName:}" failed. No retries permitted until 2026-07-18 02:39:42.643364933 +0000 UTC m=+6.247012614 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fdl8j" (UniqueName: "kubernetes.io/projected/01c7795c-7254-44d2-95eb-d3f076cd4bea-kube-api-access-fdl8j") pod "network-check-target-8rfhc" (UID: "01c7795c-7254-44d2-95eb-d3f076cd4bea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 02:39:40.859374 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:40.859023 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-17 02:34:39 +0000 UTC" deadline="2027-12-14 08:43:21.105794483 +0000 UTC" Jul 18 02:39:40.859374 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:40.859080 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12342h3m40.246718526s" Jul 18 02:39:40.898234 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:40.897451 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:39:40.898234 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:40.897592 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nht5x" podUID="870f3a38-67e6-42ea-9398-f12eeb350f5d" Jul 18 02:39:40.898234 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:40.897960 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:39:40.898234 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:40.898047 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqtcn" podUID="d8073815-9d72-4741-8447-ca9fa5dbd049" Jul 18 02:39:40.898234 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:40.898104 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:39:40.898234 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:40.898167 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rfhc" podUID="01c7795c-7254-44d2-95eb-d3f076cd4bea" Jul 18 02:39:40.927286 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:40.926458 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-4.ec2.internal" event={"ID":"10427b699bfe492315ce6d6f3f8955b6","Type":"ContainerStarted","Data":"4d34121daae3a271e897fcb43a4bcf6811b775e37857d7fc8fff4bad6d895d28"} Jul 18 02:39:40.936753 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:40.936671 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xt7md" event={"ID":"f24c6307-0556-4abf-ad93-fe954de837b4","Type":"ContainerStarted","Data":"6b0eba848ff61879da1e430e8da772d3b092dfb6fe981ab947d4d6b26a3746cd"} Jul 18 02:39:40.942990 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:40.942920 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-4.ec2.internal" podStartSLOduration=2.9429021459999998 podStartE2EDuration="2.942902146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 02:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:39:40.942308238 +0000 UTC m=+4.545955925" watchObservedRunningTime="2026-07-18 02:39:40.942902146 +0000 UTC m=+4.546549834" Jul 18 02:39:40.946595 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:40.946554 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6nsv" event={"ID":"6ee0c486-dd44-4685-9695-2dff5f42ace2","Type":"ContainerStarted","Data":"277a0cfa2f03e63443141e35a4a7ff2095c331e22e15086439ce7c374ea520b1"} Jul 18 02:39:40.959776 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:40.959746 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" event={"ID":"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18","Type":"ContainerStarted","Data":"1ed27ae7f9b232ced0f5e6c4792996fb5d87d19c9ec2fb901b96c3465666ab4b"} Jul 18 02:39:40.969472 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:40.969447 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tlhsl" event={"ID":"7321f9a8-9e03-4dff-a574-0175b0c89260","Type":"ContainerStarted","Data":"47dba9197728d8d202f1f151c038f9b13d03069363d1e75e318dd3f65180c339"} Jul 18 02:39:40.985249 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:40.985155 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvpps" event={"ID":"35cb46c9-0c8a-4562-a059-521ad01f2552","Type":"ContainerStarted","Data":"59f1532ddb5057c7cd78e8a6e1820a3a03b18ad46fcdaa86a02b7a59f4fcca74"} Jul 18 02:39:41.000034 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:40.999957 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-625jl" event={"ID":"1b59997b-ff58-43bc-b393-a6c42f6f1ed4","Type":"ContainerStarted","Data":"881e0f265fb4f30dd25672960088a05ac79304c8418e85c38a34b8c088ae7518"} Jul 18 02:39:41.006986 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:41.006962 2585 generic.go:356] "Generic (PLEG): container finished" podID="ea3598a841ad3d314209c397f6def29f" containerID="826ad56e2c2ff34e226cc67f36bf9680b96c0c6abd3886f2211d0c2e8fae5179" exitCode=0 Jul 18 02:39:41.007093 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:41.007049 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-4.ec2.internal" event={"ID":"ea3598a841ad3d314209c397f6def29f","Type":"ContainerDied","Data":"826ad56e2c2ff34e226cc67f36bf9680b96c0c6abd3886f2211d0c2e8fae5179"} Jul 18 02:39:41.012397 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:41.012374 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-82mcs" event={"ID":"6b049edb-9907-4ff4-af47-b54194c0776d","Type":"ContainerStarted","Data":"48cf66a048888f4c410f73fe459bd20ac4b9820c1861245c70aeb06b1eb86cf1"} Jul 18 02:39:41.026029 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:41.026004 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8b2g5" event={"ID":"d56b8798-4bc3-4101-b769-c129c5097167","Type":"ContainerStarted","Data":"4679e0f7e10b5c6750944a5666b56828f8ca7d261dc082b29466ac003f107e4f"} Jul 18 02:39:41.029264 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:41.029240 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h6v9b" event={"ID":"76644748-24e4-4b1d-a6b1-68887436724c","Type":"ContainerStarted","Data":"48c8f3b94786775e2c5b2cf56b2629d28e528ed9f37ec462ea66455f880530ea"} Jul 18 02:39:42.062596 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:42.062518 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-4.ec2.internal" event={"ID":"ea3598a841ad3d314209c397f6def29f","Type":"ContainerStarted","Data":"406d3f24061fd56a36ff428d07bf3a854d3003d5f8df65d5bf472627be026a8f"} Jul 18 02:39:42.460531 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:42.459617 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870f3a38-67e6-42ea-9398-f12eeb350f5d-metrics-certs\") pod \"network-metrics-daemon-nht5x\" (UID: \"870f3a38-67e6-42ea-9398-f12eeb350f5d\") " pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:39:42.460531 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:42.460049 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 02:39:42.460531 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:42.460111 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/870f3a38-67e6-42ea-9398-f12eeb350f5d-metrics-certs podName:870f3a38-67e6-42ea-9398-f12eeb350f5d nodeName:}" failed. No retries permitted until 2026-07-18 02:39:46.460094284 +0000 UTC m=+10.063741964 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/870f3a38-67e6-42ea-9398-f12eeb350f5d-metrics-certs") pod "network-metrics-daemon-nht5x" (UID: "870f3a38-67e6-42ea-9398-f12eeb350f5d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 02:39:42.561520 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:42.560882 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8073815-9d72-4741-8447-ca9fa5dbd049-original-pull-secret\") pod \"global-pull-secret-syncer-mqtcn\" (UID: \"d8073815-9d72-4741-8447-ca9fa5dbd049\") " pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:39:42.561520 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:42.561058 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 02:39:42.561520 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:42.561115 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8073815-9d72-4741-8447-ca9fa5dbd049-original-pull-secret podName:d8073815-9d72-4741-8447-ca9fa5dbd049 nodeName:}" failed. No retries permitted until 2026-07-18 02:39:46.561099289 +0000 UTC m=+10.164746969 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8073815-9d72-4741-8447-ca9fa5dbd049-original-pull-secret") pod "global-pull-secret-syncer-mqtcn" (UID: "d8073815-9d72-4741-8447-ca9fa5dbd049") : object "kube-system"/"original-pull-secret" not registered Jul 18 02:39:42.663078 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:42.662347 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdl8j\" (UniqueName: \"kubernetes.io/projected/01c7795c-7254-44d2-95eb-d3f076cd4bea-kube-api-access-fdl8j\") pod \"network-check-target-8rfhc\" (UID: \"01c7795c-7254-44d2-95eb-d3f076cd4bea\") " pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:39:42.663078 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:42.662546 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 02:39:42.663078 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:42.662592 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 02:39:42.663078 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:42.662620 2585 projected.go:196] Error preparing data for projected volume kube-api-access-fdl8j for pod openshift-network-diagnostics/network-check-target-8rfhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 02:39:42.663078 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:42.662698 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01c7795c-7254-44d2-95eb-d3f076cd4bea-kube-api-access-fdl8j podName:01c7795c-7254-44d2-95eb-d3f076cd4bea nodeName:}" failed. No retries permitted until 2026-07-18 02:39:46.662660984 +0000 UTC m=+10.266308662 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fdl8j" (UniqueName: "kubernetes.io/projected/01c7795c-7254-44d2-95eb-d3f076cd4bea-kube-api-access-fdl8j") pod "network-check-target-8rfhc" (UID: "01c7795c-7254-44d2-95eb-d3f076cd4bea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 02:39:42.895256 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:42.894569 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:39:42.895256 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:42.894716 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nht5x" podUID="870f3a38-67e6-42ea-9398-f12eeb350f5d" Jul 18 02:39:42.895256 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:42.895087 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:39:42.895256 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:42.895205 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqtcn" podUID="d8073815-9d72-4741-8447-ca9fa5dbd049" Jul 18 02:39:42.895256 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:42.895259 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:39:42.895648 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:42.895330 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rfhc" podUID="01c7795c-7254-44d2-95eb-d3f076cd4bea" Jul 18 02:39:44.894903 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:44.894861 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:39:44.895382 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:44.894982 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rfhc" podUID="01c7795c-7254-44d2-95eb-d3f076cd4bea" Jul 18 02:39:44.895382 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:44.895269 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:39:44.895494 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:44.895384 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqtcn" podUID="d8073815-9d72-4741-8447-ca9fa5dbd049" Jul 18 02:39:44.895494 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:44.895459 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:39:44.895594 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:44.895542 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nht5x" podUID="870f3a38-67e6-42ea-9398-f12eeb350f5d" Jul 18 02:39:46.493171 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:46.493135 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870f3a38-67e6-42ea-9398-f12eeb350f5d-metrics-certs\") pod \"network-metrics-daemon-nht5x\" (UID: \"870f3a38-67e6-42ea-9398-f12eeb350f5d\") " pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:39:46.493649 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:46.493284 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 02:39:46.493649 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:46.493360 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/870f3a38-67e6-42ea-9398-f12eeb350f5d-metrics-certs podName:870f3a38-67e6-42ea-9398-f12eeb350f5d nodeName:}" failed. No retries permitted until 2026-07-18 02:39:54.493341414 +0000 UTC m=+18.096989085 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/870f3a38-67e6-42ea-9398-f12eeb350f5d-metrics-certs") pod "network-metrics-daemon-nht5x" (UID: "870f3a38-67e6-42ea-9398-f12eeb350f5d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 02:39:46.594233 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:46.594203 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8073815-9d72-4741-8447-ca9fa5dbd049-original-pull-secret\") pod \"global-pull-secret-syncer-mqtcn\" (UID: \"d8073815-9d72-4741-8447-ca9fa5dbd049\") " pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:39:46.594378 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:46.594313 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 02:39:46.594378 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:46.594368 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8073815-9d72-4741-8447-ca9fa5dbd049-original-pull-secret podName:d8073815-9d72-4741-8447-ca9fa5dbd049 nodeName:}" failed. No retries permitted until 2026-07-18 02:39:54.594353767 +0000 UTC m=+18.198001431 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8073815-9d72-4741-8447-ca9fa5dbd049-original-pull-secret") pod "global-pull-secret-syncer-mqtcn" (UID: "d8073815-9d72-4741-8447-ca9fa5dbd049") : object "kube-system"/"original-pull-secret" not registered Jul 18 02:39:46.695412 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:46.695378 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdl8j\" (UniqueName: \"kubernetes.io/projected/01c7795c-7254-44d2-95eb-d3f076cd4bea-kube-api-access-fdl8j\") pod \"network-check-target-8rfhc\" (UID: \"01c7795c-7254-44d2-95eb-d3f076cd4bea\") " pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:39:46.695597 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:46.695558 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 02:39:46.695597 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:46.695581 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 02:39:46.695597 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:46.695592 2585 projected.go:196] Error preparing data for projected volume kube-api-access-fdl8j for pod openshift-network-diagnostics/network-check-target-8rfhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 02:39:46.695738 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:46.695656 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01c7795c-7254-44d2-95eb-d3f076cd4bea-kube-api-access-fdl8j podName:01c7795c-7254-44d2-95eb-d3f076cd4bea nodeName:}" failed. No retries permitted until 2026-07-18 02:39:54.695637379 +0000 UTC m=+18.299285058 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fdl8j" (UniqueName: "kubernetes.io/projected/01c7795c-7254-44d2-95eb-d3f076cd4bea-kube-api-access-fdl8j") pod "network-check-target-8rfhc" (UID: "01c7795c-7254-44d2-95eb-d3f076cd4bea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 02:39:46.895471 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:46.895340 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:39:46.895761 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:46.895385 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:39:46.895761 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:46.895403 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:39:46.896778 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:46.896069 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nht5x" podUID="870f3a38-67e6-42ea-9398-f12eeb350f5d" Jul 18 02:39:46.896778 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:46.896148 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqtcn" podUID="d8073815-9d72-4741-8447-ca9fa5dbd049" Jul 18 02:39:46.896778 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:46.895977 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rfhc" podUID="01c7795c-7254-44d2-95eb-d3f076cd4bea" Jul 18 02:39:48.894443 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:48.894411 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:39:48.894848 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:48.894546 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nht5x" podUID="870f3a38-67e6-42ea-9398-f12eeb350f5d" Jul 18 02:39:48.894948 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:48.894926 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:39:48.895048 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:48.895027 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqtcn" podUID="d8073815-9d72-4741-8447-ca9fa5dbd049" Jul 18 02:39:48.895105 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:48.895083 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:39:48.895229 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:48.895203 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rfhc" podUID="01c7795c-7254-44d2-95eb-d3f076cd4bea" Jul 18 02:39:50.894546 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:50.894521 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:39:50.894546 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:50.894537 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:39:50.894901 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:50.894644 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nht5x" podUID="870f3a38-67e6-42ea-9398-f12eeb350f5d" Jul 18 02:39:50.894901 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:50.894731 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:39:50.894901 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:50.894721 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqtcn" podUID="d8073815-9d72-4741-8447-ca9fa5dbd049" Jul 18 02:39:50.894901 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:50.894813 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rfhc" podUID="01c7795c-7254-44d2-95eb-d3f076cd4bea" Jul 18 02:39:51.080217 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:51.080164 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h6v9b" event={"ID":"76644748-24e4-4b1d-a6b1-68887436724c","Type":"ContainerStarted","Data":"59bd702c886daf5a6d94a68df3f515e74722dbf578bd2e4d53ecac0790d32184"} Jul 18 02:39:51.082078 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:51.082049 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6nsv" event={"ID":"6ee0c486-dd44-4685-9695-2dff5f42ace2","Type":"ContainerStarted","Data":"d620cf3da765c9b8bcf6fad86ce9c2acbca5b4131cfb34eda5e4c8d2edb99b6c"} Jul 18 02:39:51.083685 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:51.083614 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-625jl" event={"ID":"1b59997b-ff58-43bc-b393-a6c42f6f1ed4","Type":"ContainerStarted","Data":"81dd5dfcc64104f36132a1539bc08b2e9b1b0f8f29cf17d8e2001c0063165703"} Jul 18 02:39:51.094334 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:51.094273 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-4.ec2.internal" podStartSLOduration=13.094255736 podStartE2EDuration="13.094255736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 02:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:39:42.077554923 +0000 UTC m=+5.681202610" watchObservedRunningTime="2026-07-18 02:39:51.094255736 +0000 UTC m=+14.697903430" Jul 18 02:39:51.106942 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:51.106897 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-h6v9b" podStartSLOduration=3.387581758 podStartE2EDuration="14.106881084s" totalImagesPullingTime="10.719299326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:39:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:39:51.094044871 +0000 UTC m=+14.697692558" watchObservedRunningTime="2026-07-18 02:39:51.106881084 +0000 UTC m=+14.710528771" Jul 18 02:39:51.107127 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:51.107097 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-625jl" podStartSLOduration=3.32968495 podStartE2EDuration="14.107091261s" totalImagesPullingTime="10.777406311s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 02:39:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:39:51.106803892 +0000 UTC m=+14.710451582" watchObservedRunningTime="2026-07-18 02:39:51.107091261 +0000 UTC m=+14.710738947" Jul 18 02:39:52.306194 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:52.306159 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-h6v9b" Jul 18 02:39:52.306784 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:52.306766 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-h6v9b" Jul 18 02:39:52.894193 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:52.894148 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:39:52.894193 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:52.894149 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:39:52.894418 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:52.894267 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:39:52.894418 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:52.894276 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nht5x" podUID="870f3a38-67e6-42ea-9398-f12eeb350f5d" Jul 18 02:39:52.894418 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:52.894345 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rfhc" podUID="01c7795c-7254-44d2-95eb-d3f076cd4bea" Jul 18 02:39:52.894570 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:52.894416 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqtcn" podUID="d8073815-9d72-4741-8447-ca9fa5dbd049" Jul 18 02:39:53.089268 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:53.089223 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xt7md" event={"ID":"f24c6307-0556-4abf-ad93-fe954de837b4","Type":"ContainerStarted","Data":"2a92f82be3d8160b2b9cadbcef4fad2bf2bf82d9ea7222a2b4ecbba7a9dd1f7f"} Jul 18 02:39:54.091345 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:54.091146 2585 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 18 02:39:54.563566 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:54.563532 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870f3a38-67e6-42ea-9398-f12eeb350f5d-metrics-certs\") pod \"network-metrics-daemon-nht5x\" (UID: \"870f3a38-67e6-42ea-9398-f12eeb350f5d\") " pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:39:54.563733 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:54.563676 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 02:39:54.563789 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:54.563750 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/870f3a38-67e6-42ea-9398-f12eeb350f5d-metrics-certs podName:870f3a38-67e6-42ea-9398-f12eeb350f5d nodeName:}" failed. No retries permitted until 2026-07-18 02:40:10.563729335 +0000 UTC m=+34.167377015 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/870f3a38-67e6-42ea-9398-f12eeb350f5d-metrics-certs") pod "network-metrics-daemon-nht5x" (UID: "870f3a38-67e6-42ea-9398-f12eeb350f5d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 02:39:54.663966 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:54.663933 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8073815-9d72-4741-8447-ca9fa5dbd049-original-pull-secret\") pod \"global-pull-secret-syncer-mqtcn\" (UID: \"d8073815-9d72-4741-8447-ca9fa5dbd049\") " pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:39:54.664143 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:54.664071 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 02:39:54.664143 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:54.664139 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8073815-9d72-4741-8447-ca9fa5dbd049-original-pull-secret podName:d8073815-9d72-4741-8447-ca9fa5dbd049 nodeName:}" failed. No retries permitted until 2026-07-18 02:40:10.664121421 +0000 UTC m=+34.267769085 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8073815-9d72-4741-8447-ca9fa5dbd049-original-pull-secret") pod "global-pull-secret-syncer-mqtcn" (UID: "d8073815-9d72-4741-8447-ca9fa5dbd049") : object "kube-system"/"original-pull-secret" not registered Jul 18 02:39:54.764589 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:54.764562 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdl8j\" (UniqueName: \"kubernetes.io/projected/01c7795c-7254-44d2-95eb-d3f076cd4bea-kube-api-access-fdl8j\") pod \"network-check-target-8rfhc\" (UID: \"01c7795c-7254-44d2-95eb-d3f076cd4bea\") " pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:39:54.764764 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:54.764700 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 02:39:54.764764 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:54.764721 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 02:39:54.764764 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:54.764732 2585 projected.go:196] Error preparing data for projected volume kube-api-access-fdl8j for pod openshift-network-diagnostics/network-check-target-8rfhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 02:39:54.764920 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:54.764792 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01c7795c-7254-44d2-95eb-d3f076cd4bea-kube-api-access-fdl8j podName:01c7795c-7254-44d2-95eb-d3f076cd4bea nodeName:}" failed. No retries permitted until 2026-07-18 02:40:10.764775115 +0000 UTC m=+34.368422798 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fdl8j" (UniqueName: "kubernetes.io/projected/01c7795c-7254-44d2-95eb-d3f076cd4bea-kube-api-access-fdl8j") pod "network-check-target-8rfhc" (UID: "01c7795c-7254-44d2-95eb-d3f076cd4bea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 02:39:54.895099 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:54.895029 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:39:54.895099 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:54.895050 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:39:54.895324 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:54.895038 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:39:54.895324 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:54.895158 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nht5x" podUID="870f3a38-67e6-42ea-9398-f12eeb350f5d" Jul 18 02:39:54.895324 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:54.895272 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqtcn" podUID="d8073815-9d72-4741-8447-ca9fa5dbd049" Jul 18 02:39:54.895460 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:54.895341 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rfhc" podUID="01c7795c-7254-44d2-95eb-d3f076cd4bea" Jul 18 02:39:56.895616 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:56.895587 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:39:56.896242 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:56.895694 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nht5x" podUID="870f3a38-67e6-42ea-9398-f12eeb350f5d" Jul 18 02:39:56.896242 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:56.895774 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:39:56.896242 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:56.895879 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqtcn" podUID="d8073815-9d72-4741-8447-ca9fa5dbd049" Jul 18 02:39:56.896242 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:56.895921 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:39:56.896242 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:56.895997 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rfhc" podUID="01c7795c-7254-44d2-95eb-d3f076cd4bea" Jul 18 02:39:58.895268 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:58.895101 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:39:58.895736 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:58.895101 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:39:58.895736 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:58.895354 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nht5x" podUID="870f3a38-67e6-42ea-9398-f12eeb350f5d" Jul 18 02:39:58.895736 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:58.895438 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqtcn" podUID="d8073815-9d72-4741-8447-ca9fa5dbd049" Jul 18 02:39:58.895736 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:58.895166 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:39:58.895736 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:39:58.895529 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rfhc" podUID="01c7795c-7254-44d2-95eb-d3f076cd4bea" Jul 18 02:39:59.102080 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:59.101821 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tlhsl" event={"ID":"7321f9a8-9e03-4dff-a574-0175b0c89260","Type":"ContainerStarted","Data":"c999dfa90b45859a60f1a624b1d1af6e255932276586aa838ee1814b6718ebcc"} Jul 18 02:39:59.103503 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:59.103472 2585 generic.go:356] "Generic (PLEG): container finished" podID="35cb46c9-0c8a-4562-a059-521ad01f2552" containerID="dcfd3f25ceed5474bfe2c1d82da3deb2cce5d3cbf53b34fd040cdb7aaf0fd2a4" exitCode=0 Jul 18 02:39:59.103634 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:59.103557 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvpps" event={"ID":"35cb46c9-0c8a-4562-a059-521ad01f2552","Type":"ContainerDied","Data":"dcfd3f25ceed5474bfe2c1d82da3deb2cce5d3cbf53b34fd040cdb7aaf0fd2a4"} Jul 18 02:39:59.105221 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:59.105146 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-82mcs" event={"ID":"6b049edb-9907-4ff4-af47-b54194c0776d","Type":"ContainerStarted","Data":"7ebfc3a8edfc0e6b9ba7f293e1806c0f4ca6b6c3e4c937ba7b9ecb688abb4bd1"} Jul 18 02:39:59.106724 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:59.106702 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8b2g5" event={"ID":"d56b8798-4bc3-4101-b769-c129c5097167","Type":"ContainerStarted","Data":"40fb63efa35baf7e3ac3a65bcfe6480a0ee3af9cfc8006ff968c587f93a2746d"} Jul 18 02:39:59.109654 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:59.109631 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" event={"ID":"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18","Type":"ContainerStarted","Data":"3763c12f9d94fb03af373ab256704b81b5834b2312b326d3c726481e9102e3c9"} Jul 18 02:39:59.109726 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:59.109707 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" event={"ID":"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18","Type":"ContainerStarted","Data":"6134256cd917234681232c356fe64fe44519aea099600576509620a904b1537e"} Jul 18 02:39:59.109775 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:59.109755 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" event={"ID":"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18","Type":"ContainerStarted","Data":"399c82a52fbeb2b7c0e6631059d23378a0da3324de3ab1b4b6d154fef84c691b"} Jul 18 02:39:59.109807 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:59.109801 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" event={"ID":"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18","Type":"ContainerStarted","Data":"fc853fe511ae157712eb014c6831d8922d171f1603ef6bcf8c2fd72ccd18568a"} Jul 18 02:39:59.109861 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:59.109845 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" event={"ID":"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18","Type":"ContainerStarted","Data":"1b925847ec14c29772a242e6b6c37b8d88b8e96902d1d83bfb1556b227b13ee3"} Jul 18 02:39:59.116567 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:59.116523 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tlhsl" podStartSLOduration=11.340920836 podStartE2EDuration="22.116475622s" totalImagesPullingTime="10.775554786s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 02:39:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:39:59.116292991 +0000 UTC m=+22.719940676" watchObservedRunningTime="2026-07-18 02:39:59.116475622 +0000 UTC m=+22.720123308" Jul 18 02:39:59.116797 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:59.116773 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xt7md" podStartSLOduration=11.348839406 podStartE2EDuration="22.116765499s" totalImagesPullingTime="10.767926093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 02:39:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:39:53.103561103 +0000 UTC m=+16.707208790" watchObservedRunningTime="2026-07-18 02:39:59.116765499 +0000 UTC m=+22.720413188" Jul 18 02:39:59.131740 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:59.131700 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-82mcs" podStartSLOduration=11.299001109 podStartE2EDuration="22.131685476s" totalImagesPullingTime="10.832684367s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 02:39:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:39:59.131656466 +0000 UTC m=+22.735304152" watchObservedRunningTime="2026-07-18 02:39:59.131685476 +0000 UTC m=+22.735333165" Jul 18 02:39:59.181174 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:59.181127 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8b2g5" podStartSLOduration=3.60483522 podStartE2EDuration="22.181115852s" totalImagesPullingTime="18.576280632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 02:39:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:39:59.18060585 +0000 UTC m=+22.784253536" watchObservedRunningTime="2026-07-18 02:39:59.181115852 +0000 UTC m=+22.784763543" Jul 18 02:39:59.656595 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:59.656573 2585 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 18 02:39:59.875531 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:59.875407 2585 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-18T02:39:59.656591359Z","UUID":"163ebd80-63ae-442f-aa58-46250a295345","Handler":null,"Name":"","Endpoint":""} Jul 18 02:39:59.878471 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:59.878451 2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 18 02:39:59.878591 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:39:59.878488 2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 18 02:40:00.113953 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:00.113913 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6nsv" event={"ID":"6ee0c486-dd44-4685-9695-2dff5f42ace2","Type":"ContainerStarted","Data":"a72a79ce5982e3c8b2846ba44e5fea16cd68389eeb9ca9456b49dab416df6376"} Jul 18 02:40:00.117399 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:00.117374 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" event={"ID":"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18","Type":"ContainerStarted","Data":"e083dc02fb29e5df4054a3200e8dbffe1f4351579f99487250a2c25c1ca7e631"} Jul 18 02:40:00.894540 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:00.894505 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:40:00.894758 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:00.894505 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:40:00.894758 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:00.894622 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rfhc" podUID="01c7795c-7254-44d2-95eb-d3f076cd4bea" Jul 18 02:40:00.894758 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:00.894508 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:40:00.894758 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:00.894677 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nht5x" podUID="870f3a38-67e6-42ea-9398-f12eeb350f5d" Jul 18 02:40:00.894893 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:00.894753 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqtcn" podUID="d8073815-9d72-4741-8447-ca9fa5dbd049" Jul 18 02:40:02.124237 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:02.124200 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6nsv" event={"ID":"6ee0c486-dd44-4685-9695-2dff5f42ace2","Type":"ContainerStarted","Data":"146a17707fc84860280a7ee7ca5f87f53a3dc182126779cb2650ef82b12cc14c"} Jul 18 02:40:02.127090 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:02.127067 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" event={"ID":"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18","Type":"ContainerStarted","Data":"2827952e5a9d8153078756daea47ab63e838d4570a1eb35e2764d7b46e718e22"} Jul 18 02:40:02.141824 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:02.141781 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6nsv" podStartSLOduration=4.431423401 podStartE2EDuration="25.141765258s" totalImagesPullingTime="20.710341857s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 02:39:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:40:02.140140295 +0000 UTC m=+25.743787983" watchObservedRunningTime="2026-07-18 02:40:02.141765258 +0000 UTC m=+25.745412945" Jul 18 02:40:02.858165 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:02.858133 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-h6v9b" Jul 18 02:40:02.858353 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:02.858279 2585 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 18 02:40:02.858700 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:02.858676 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h6v9b" Jul 18 02:40:02.894146 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:02.894120 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:40:02.894257 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:02.894120 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:40:02.894257 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:02.894228 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqtcn" podUID="d8073815-9d72-4741-8447-ca9fa5dbd049" Jul 18 02:40:02.894330 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:02.894127 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:40:02.894330 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:02.894275 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nht5x" podUID="870f3a38-67e6-42ea-9398-f12eeb350f5d" Jul 18 02:40:02.894389 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:02.894349 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rfhc" podUID="01c7795c-7254-44d2-95eb-d3f076cd4bea" Jul 18 02:40:04.894470 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:04.894442 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:40:04.894980 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:04.894442 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:40:04.894980 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:04.894560 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nht5x" podUID="870f3a38-67e6-42ea-9398-f12eeb350f5d" Jul 18 02:40:04.894980 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:04.894455 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:40:04.894980 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:04.894598 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqtcn" podUID="d8073815-9d72-4741-8447-ca9fa5dbd049" Jul 18 02:40:04.894980 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:04.894673 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rfhc" podUID="01c7795c-7254-44d2-95eb-d3f076cd4bea" Jul 18 02:40:05.137027 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:05.136903 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" event={"ID":"6b57a8b2-d5b6-4cce-9fb0-a3746be0de18","Type":"ContainerStarted","Data":"cc1cfe417162af18c4761ef756c7c683f141f85d0422691c9a2d710415a4a616"} Jul 18 02:40:05.137285 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:05.137266 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:40:05.137396 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:05.137365 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:40:05.137477 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:05.137415 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:40:05.152616 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:05.152525 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:40:05.152978 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:05.152950 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:40:05.164000 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:05.163961 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" podStartSLOduration=9.574061593 podStartE2EDuration="28.16394753s" totalImagesPullingTime="18.589885937s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 02:39:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:40:05.163915245 +0000 UTC m=+28.767562932" watchObservedRunningTime="2026-07-18 02:40:05.16394753 +0000 UTC m=+28.767595207" Jul 18 02:40:06.140382 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:06.140345 2585 generic.go:356] "Generic (PLEG): container finished" podID="35cb46c9-0c8a-4562-a059-521ad01f2552" containerID="a50918292c276064aadaee37d7e6a896e5c623410a9757ca83b148ede73282d6" exitCode=0 Jul 18 02:40:06.140963 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:06.140425 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvpps" event={"ID":"35cb46c9-0c8a-4562-a059-521ad01f2552","Type":"ContainerDied","Data":"a50918292c276064aadaee37d7e6a896e5c623410a9757ca83b148ede73282d6"} Jul 18 02:40:06.895262 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:06.895235 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:40:06.895433 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:06.895320 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:40:06.895433 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:06.895354 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqtcn" podUID="d8073815-9d72-4741-8447-ca9fa5dbd049" Jul 18 02:40:06.895433 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:06.895367 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:40:06.895550 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:06.895429 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rfhc" podUID="01c7795c-7254-44d2-95eb-d3f076cd4bea" Jul 18 02:40:06.895550 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:06.895533 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nht5x" podUID="870f3a38-67e6-42ea-9398-f12eeb350f5d" Jul 18 02:40:06.952784 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:06.952753 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mqtcn"] Jul 18 02:40:06.956220 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:06.956176 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8rfhc"] Jul 18 02:40:06.957003 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:06.956979 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nht5x"] Jul 18 02:40:07.142546 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:07.142524 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:40:07.142880 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:07.142546 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:40:07.142880 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:07.142532 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:40:07.142880 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:07.142616 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqtcn" podUID="d8073815-9d72-4741-8447-ca9fa5dbd049" Jul 18 02:40:07.143042 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:07.142937 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nht5x" podUID="870f3a38-67e6-42ea-9398-f12eeb350f5d" Jul 18 02:40:07.143042 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:07.143024 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rfhc" podUID="01c7795c-7254-44d2-95eb-d3f076cd4bea" Jul 18 02:40:08.147073 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:08.146934 2585 generic.go:356] "Generic (PLEG): container finished" podID="35cb46c9-0c8a-4562-a059-521ad01f2552" containerID="83d5f97e1f98a12a6662760a79c9603469a60685dcad7fb9b508d65d16073cb8" exitCode=0 Jul 18 02:40:08.147450 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:08.147010 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvpps" event={"ID":"35cb46c9-0c8a-4562-a059-521ad01f2552","Type":"ContainerDied","Data":"83d5f97e1f98a12a6662760a79c9603469a60685dcad7fb9b508d65d16073cb8"} Jul 18 02:40:08.894860 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:08.894829 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:40:08.895022 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:08.894829 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:40:08.895022 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:08.894945 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nht5x" podUID="870f3a38-67e6-42ea-9398-f12eeb350f5d" Jul 18 02:40:08.895022 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:08.894959 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:40:08.895022 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:08.895007 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqtcn" podUID="d8073815-9d72-4741-8447-ca9fa5dbd049" Jul 18 02:40:08.895171 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:08.895057 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rfhc" podUID="01c7795c-7254-44d2-95eb-d3f076cd4bea" Jul 18 02:40:09.151001 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:09.150943 2585 generic.go:356] "Generic (PLEG): container finished" podID="35cb46c9-0c8a-4562-a059-521ad01f2552" containerID="6b2a0252713e193eb32a6cf4b3efe5a0f0ca1a01f4680eeb8bdec6d0b91a7b77" exitCode=0 Jul 18 02:40:09.151277 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:09.150998 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvpps" event={"ID":"35cb46c9-0c8a-4562-a059-521ad01f2552","Type":"ContainerDied","Data":"6b2a0252713e193eb32a6cf4b3efe5a0f0ca1a01f4680eeb8bdec6d0b91a7b77"} Jul 18 02:40:10.587265 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:10.587229 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870f3a38-67e6-42ea-9398-f12eeb350f5d-metrics-certs\") pod \"network-metrics-daemon-nht5x\" (UID: \"870f3a38-67e6-42ea-9398-f12eeb350f5d\") " pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:40:10.587770 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:10.587359 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 02:40:10.587770 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:10.587421 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/870f3a38-67e6-42ea-9398-f12eeb350f5d-metrics-certs podName:870f3a38-67e6-42ea-9398-f12eeb350f5d nodeName:}" failed. No retries permitted until 2026-07-18 02:40:42.587401169 +0000 UTC m=+66.191048834 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/870f3a38-67e6-42ea-9398-f12eeb350f5d-metrics-certs") pod "network-metrics-daemon-nht5x" (UID: "870f3a38-67e6-42ea-9398-f12eeb350f5d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 02:40:10.688278 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:10.688251 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8073815-9d72-4741-8447-ca9fa5dbd049-original-pull-secret\") pod \"global-pull-secret-syncer-mqtcn\" (UID: \"d8073815-9d72-4741-8447-ca9fa5dbd049\") " pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:40:10.688417 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:10.688363 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 02:40:10.688417 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:10.688415 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8073815-9d72-4741-8447-ca9fa5dbd049-original-pull-secret podName:d8073815-9d72-4741-8447-ca9fa5dbd049 nodeName:}" failed. No retries permitted until 2026-07-18 02:40:42.688398573 +0000 UTC m=+66.292046251 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8073815-9d72-4741-8447-ca9fa5dbd049-original-pull-secret") pod "global-pull-secret-syncer-mqtcn" (UID: "d8073815-9d72-4741-8447-ca9fa5dbd049") : object "kube-system"/"original-pull-secret" not registered Jul 18 02:40:10.789303 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:10.789273 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdl8j\" (UniqueName: \"kubernetes.io/projected/01c7795c-7254-44d2-95eb-d3f076cd4bea-kube-api-access-fdl8j\") pod \"network-check-target-8rfhc\" (UID: \"01c7795c-7254-44d2-95eb-d3f076cd4bea\") " pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:40:10.789443 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:10.789408 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 02:40:10.789443 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:10.789423 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 02:40:10.789443 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:10.789431 2585 projected.go:196] Error preparing data for projected volume kube-api-access-fdl8j for pod openshift-network-diagnostics/network-check-target-8rfhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 02:40:10.789600 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:10.789488 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01c7795c-7254-44d2-95eb-d3f076cd4bea-kube-api-access-fdl8j podName:01c7795c-7254-44d2-95eb-d3f076cd4bea nodeName:}" failed. No retries permitted until 2026-07-18 02:40:42.789468389 +0000 UTC m=+66.393116071 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fdl8j" (UniqueName: "kubernetes.io/projected/01c7795c-7254-44d2-95eb-d3f076cd4bea-kube-api-access-fdl8j") pod "network-check-target-8rfhc" (UID: "01c7795c-7254-44d2-95eb-d3f076cd4bea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 02:40:10.894748 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:10.894669 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:40:10.894748 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:10.894700 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:40:10.894945 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:10.894672 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:40:10.894945 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:10.894783 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqtcn" podUID="d8073815-9d72-4741-8447-ca9fa5dbd049" Jul 18 02:40:10.894945 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:10.894876 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nht5x" podUID="870f3a38-67e6-42ea-9398-f12eeb350f5d" Jul 18 02:40:10.895096 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:10.894946 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rfhc" podUID="01c7795c-7254-44d2-95eb-d3f076cd4bea" Jul 18 02:40:11.145884 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.145815 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-4.ec2.internal" event="NodeReady" Jul 18 02:40:11.146031 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.145940 2585 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 18 02:40:11.181146 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.181117 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8"] Jul 18 02:40:11.185778 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.185750 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-695bfc9c5-tgcx7"] Jul 18 02:40:11.185915 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.185894 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" Jul 18 02:40:11.188878 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.188760 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 18 02:40:11.188878 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.188807 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 18 02:40:11.189233 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.189207 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 18 02:40:11.189528 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.189511 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 18 02:40:11.189618 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.189537 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 18 02:40:11.189782 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.189763 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 18 02:40:11.189866 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.189791 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 18 02:40:11.190523 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.190498 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58fdcb5c84-jwllm"] Jul 18 02:40:11.190645 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.190625 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695bfc9c5-tgcx7" Jul 18 02:40:11.191293 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.191275 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0825f880-ea1c-4683-b9b1-2e9aaeee31d8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75c8f6f455-bvsp8\" (UID: \"0825f880-ea1c-4683-b9b1-2e9aaeee31d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" Jul 18 02:40:11.191392 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.191306 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9hll\" (UniqueName: \"kubernetes.io/projected/0825f880-ea1c-4683-b9b1-2e9aaeee31d8-kube-api-access-n9hll\") pod \"cluster-proxy-proxy-agent-75c8f6f455-bvsp8\" (UID: \"0825f880-ea1c-4683-b9b1-2e9aaeee31d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" Jul 18 02:40:11.191392 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.191333 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0825f880-ea1c-4683-b9b1-2e9aaeee31d8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75c8f6f455-bvsp8\" (UID: \"0825f880-ea1c-4683-b9b1-2e9aaeee31d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" Jul 18 02:40:11.191470 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.191407 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0825f880-ea1c-4683-b9b1-2e9aaeee31d8-hub\") pod \"cluster-proxy-proxy-agent-75c8f6f455-bvsp8\" (UID: \"0825f880-ea1c-4683-b9b1-2e9aaeee31d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" Jul 18 02:40:11.191470 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.191455 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0825f880-ea1c-4683-b9b1-2e9aaeee31d8-ca\") pod \"cluster-proxy-proxy-agent-75c8f6f455-bvsp8\" (UID: \"0825f880-ea1c-4683-b9b1-2e9aaeee31d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" Jul 18 02:40:11.191542 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.191492 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0825f880-ea1c-4683-b9b1-2e9aaeee31d8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75c8f6f455-bvsp8\" (UID: \"0825f880-ea1c-4683-b9b1-2e9aaeee31d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" Jul 18 02:40:11.194623 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.194288 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp"] Jul 18 02:40:11.194623 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.194391 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58fdcb5c84-jwllm" Jul 18 02:40:11.195071 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.195050 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 18 02:40:11.197615 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.197593 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 18 02:40:11.197615 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.197610 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-k85hc\"" Jul 18 02:40:11.198105 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.198088 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w4wvr"] Jul 18 02:40:11.198267 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.198249 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:40:11.200798 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.200782 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7gcbm\"" Jul 18 02:40:11.200881 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.200813 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 18 02:40:11.200881 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.200815 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 18 02:40:11.200881 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.200873 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 18 02:40:11.201781 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.201765 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-695bfc9c5-tgcx7"] Jul 18 02:40:11.201852 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.201794 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8"] Jul 18 02:40:11.201852 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.201850 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w4wvr" Jul 18 02:40:11.205515 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.205469 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58fdcb5c84-jwllm"] Jul 18 02:40:11.206825 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.206787 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 18 02:40:11.206825 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.206796 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 18 02:40:11.206825 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.206807 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4pdzm\"" Jul 18 02:40:11.207018 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.206911 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vbsgm"] Jul 18 02:40:11.207353 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.207157 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 18 02:40:11.210606 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.210587 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp"] Jul 18 02:40:11.210709 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.210675 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vbsgm" Jul 18 02:40:11.211954 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.211905 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w4wvr"] Jul 18 02:40:11.213410 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.213396 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pjzzd\"" Jul 18 02:40:11.213501 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.213439 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 18 02:40:11.213640 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.213623 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 18 02:40:11.213702 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.213638 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 18 02:40:11.225365 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.225344 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vbsgm"] Jul 18 02:40:11.292606 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.292576 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0825f880-ea1c-4683-b9b1-2e9aaeee31d8-ca\") pod \"cluster-proxy-proxy-agent-75c8f6f455-bvsp8\" (UID: \"0825f880-ea1c-4683-b9b1-2e9aaeee31d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" Jul 18 02:40:11.292729 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.292618 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b893a4ff-cd96-48b6-958f-a7ac7b736143-installation-pull-secrets\") pod \"image-registry-6b7d8ff6c9-qz7qp\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:40:11.292729 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.292646 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5qt5\" (UniqueName: \"kubernetes.io/projected/2f3ca073-4807-44e8-9e0e-1b9b8850bd6f-kube-api-access-v5qt5\") pod \"managed-serviceaccount-addon-agent-58fdcb5c84-jwllm\" (UID: \"2f3ca073-4807-44e8-9e0e-1b9b8850bd6f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58fdcb5c84-jwllm" Jul 18 02:40:11.292729 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.292670 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e95a777-c662-4a6a-bac1-9fa079e5914c-metrics-tls\") pod \"dns-default-w4wvr\" (UID: \"7e95a777-c662-4a6a-bac1-9fa079e5914c\") " pod="openshift-dns/dns-default-w4wvr" Jul 18 02:40:11.292729 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.292711 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b893a4ff-cd96-48b6-958f-a7ac7b736143-trusted-ca\") pod \"image-registry-6b7d8ff6c9-qz7qp\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:40:11.292941 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.292735 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5fa2fe3a-5e31-47f9-980e-ee1ee96925d9-klusterlet-config\") pod \"klusterlet-addon-workmgr-695bfc9c5-tgcx7\" (UID: \"5fa2fe3a-5e31-47f9-980e-ee1ee96925d9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695bfc9c5-tgcx7" Jul 18 02:40:11.292941 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.292791 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0825f880-ea1c-4683-b9b1-2e9aaeee31d8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75c8f6f455-bvsp8\" (UID: \"0825f880-ea1c-4683-b9b1-2e9aaeee31d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" Jul 18 02:40:11.292941 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.292828 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-tls\") pod \"image-registry-6b7d8ff6c9-qz7qp\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:40:11.292941 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.292869 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-bound-sa-token\") pod \"image-registry-6b7d8ff6c9-qz7qp\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:40:11.293119 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.292969 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnsh4\" (UniqueName: \"kubernetes.io/projected/5fa2fe3a-5e31-47f9-980e-ee1ee96925d9-kube-api-access-gnsh4\") pod \"klusterlet-addon-workmgr-695bfc9c5-tgcx7\" (UID: \"5fa2fe3a-5e31-47f9-980e-ee1ee96925d9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695bfc9c5-tgcx7" Jul 18 02:40:11.293119 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.292995 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2f3ca073-4807-44e8-9e0e-1b9b8850bd6f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-58fdcb5c84-jwllm\" (UID: \"2f3ca073-4807-44e8-9e0e-1b9b8850bd6f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58fdcb5c84-jwllm" Jul 18 02:40:11.293119 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.293018 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2aceb1e5-bf1c-4344-aaa0-0e3f9fd11ab5-cert\") pod \"ingress-canary-vbsgm\" (UID: \"2aceb1e5-bf1c-4344-aaa0-0e3f9fd11ab5\") " pod="openshift-ingress-canary/ingress-canary-vbsgm" Jul 18 02:40:11.293119 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.293046 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx299\" (UniqueName: \"kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-kube-api-access-dx299\") pod \"image-registry-6b7d8ff6c9-qz7qp\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:40:11.293119 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.293069 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e95a777-c662-4a6a-bac1-9fa079e5914c-tmp-dir\") pod \"dns-default-w4wvr\" (UID: \"7e95a777-c662-4a6a-bac1-9fa079e5914c\") " pod="openshift-dns/dns-default-w4wvr" Jul 18 02:40:11.293369 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.293143 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0825f880-ea1c-4683-b9b1-2e9aaeee31d8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75c8f6f455-bvsp8\" (UID: \"0825f880-ea1c-4683-b9b1-2e9aaeee31d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" Jul 18 02:40:11.293369 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.293171 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9hll\" (UniqueName: \"kubernetes.io/projected/0825f880-ea1c-4683-b9b1-2e9aaeee31d8-kube-api-access-n9hll\") pod \"cluster-proxy-proxy-agent-75c8f6f455-bvsp8\" (UID: \"0825f880-ea1c-4683-b9b1-2e9aaeee31d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" Jul 18 02:40:11.293369 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.293214 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7e95a777-c662-4a6a-bac1-9fa079e5914c-config-volume\") pod \"dns-default-w4wvr\" (UID: \"7e95a777-c662-4a6a-bac1-9fa079e5914c\") " pod="openshift-dns/dns-default-w4wvr" Jul 18 02:40:11.293369 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.293242 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0825f880-ea1c-4683-b9b1-2e9aaeee31d8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75c8f6f455-bvsp8\" (UID: \"0825f880-ea1c-4683-b9b1-2e9aaeee31d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" Jul 18 02:40:11.293369 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.293283 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-certificates\") pod \"image-registry-6b7d8ff6c9-qz7qp\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:40:11.293369 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.293313 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b893a4ff-cd96-48b6-958f-a7ac7b736143-image-registry-private-configuration\") pod \"image-registry-6b7d8ff6c9-qz7qp\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:40:11.293369 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.293339 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b893a4ff-cd96-48b6-958f-a7ac7b736143-ca-trust-extracted\") pod \"image-registry-6b7d8ff6c9-qz7qp\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:40:11.293369 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.293361 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fa2fe3a-5e31-47f9-980e-ee1ee96925d9-tmp\") pod \"klusterlet-addon-workmgr-695bfc9c5-tgcx7\" (UID: \"5fa2fe3a-5e31-47f9-980e-ee1ee96925d9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695bfc9c5-tgcx7" Jul 18 02:40:11.293705 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.293390 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0825f880-ea1c-4683-b9b1-2e9aaeee31d8-hub\") pod \"cluster-proxy-proxy-agent-75c8f6f455-bvsp8\" (UID: \"0825f880-ea1c-4683-b9b1-2e9aaeee31d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" Jul 18 02:40:11.293705 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.293415 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9bb7\" (UniqueName: \"kubernetes.io/projected/7e95a777-c662-4a6a-bac1-9fa079e5914c-kube-api-access-z9bb7\") pod \"dns-default-w4wvr\" (UID: \"7e95a777-c662-4a6a-bac1-9fa079e5914c\") " pod="openshift-dns/dns-default-w4wvr" Jul 18 02:40:11.293705 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.293441 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh9dq\" (UniqueName: \"kubernetes.io/projected/2aceb1e5-bf1c-4344-aaa0-0e3f9fd11ab5-kube-api-access-hh9dq\") pod \"ingress-canary-vbsgm\" (UID: \"2aceb1e5-bf1c-4344-aaa0-0e3f9fd11ab5\") " pod="openshift-ingress-canary/ingress-canary-vbsgm" Jul 18 02:40:11.293897 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.293873 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0825f880-ea1c-4683-b9b1-2e9aaeee31d8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75c8f6f455-bvsp8\" (UID: \"0825f880-ea1c-4683-b9b1-2e9aaeee31d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" Jul 18 02:40:11.297496 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.297453 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0825f880-ea1c-4683-b9b1-2e9aaeee31d8-ca\") pod \"cluster-proxy-proxy-agent-75c8f6f455-bvsp8\" (UID: \"0825f880-ea1c-4683-b9b1-2e9aaeee31d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" Jul 18 02:40:11.297496 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.297462 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0825f880-ea1c-4683-b9b1-2e9aaeee31d8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75c8f6f455-bvsp8\" (UID: \"0825f880-ea1c-4683-b9b1-2e9aaeee31d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" Jul 18 02:40:11.297629 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.297569 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0825f880-ea1c-4683-b9b1-2e9aaeee31d8-hub\") pod \"cluster-proxy-proxy-agent-75c8f6f455-bvsp8\" (UID: \"0825f880-ea1c-4683-b9b1-2e9aaeee31d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" Jul 18 02:40:11.297629 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.297595 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0825f880-ea1c-4683-b9b1-2e9aaeee31d8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75c8f6f455-bvsp8\" (UID: \"0825f880-ea1c-4683-b9b1-2e9aaeee31d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" Jul 18 02:40:11.301469 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.301449 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9hll\" (UniqueName: \"kubernetes.io/projected/0825f880-ea1c-4683-b9b1-2e9aaeee31d8-kube-api-access-n9hll\") pod \"cluster-proxy-proxy-agent-75c8f6f455-bvsp8\" (UID: \"0825f880-ea1c-4683-b9b1-2e9aaeee31d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" Jul 18 02:40:11.394202 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.394164 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7e95a777-c662-4a6a-bac1-9fa079e5914c-config-volume\") pod \"dns-default-w4wvr\" (UID: \"7e95a777-c662-4a6a-bac1-9fa079e5914c\") " pod="openshift-dns/dns-default-w4wvr" Jul 18 02:40:11.394300 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.394218 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-certificates\") pod \"image-registry-6b7d8ff6c9-qz7qp\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:40:11.394300 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.394243 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b893a4ff-cd96-48b6-958f-a7ac7b736143-image-registry-private-configuration\") pod \"image-registry-6b7d8ff6c9-qz7qp\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:40:11.394300 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.394259 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b893a4ff-cd96-48b6-958f-a7ac7b736143-ca-trust-extracted\") pod \"image-registry-6b7d8ff6c9-qz7qp\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:40:11.394300 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.394273 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fa2fe3a-5e31-47f9-980e-ee1ee96925d9-tmp\") pod \"klusterlet-addon-workmgr-695bfc9c5-tgcx7\" (UID: \"5fa2fe3a-5e31-47f9-980e-ee1ee96925d9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695bfc9c5-tgcx7" Jul 18 02:40:11.394300 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.394290 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9bb7\" (UniqueName: \"kubernetes.io/projected/7e95a777-c662-4a6a-bac1-9fa079e5914c-kube-api-access-z9bb7\") pod \"dns-default-w4wvr\" (UID: \"7e95a777-c662-4a6a-bac1-9fa079e5914c\") " pod="openshift-dns/dns-default-w4wvr" Jul 18 02:40:11.394548 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.394307 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh9dq\" (UniqueName: \"kubernetes.io/projected/2aceb1e5-bf1c-4344-aaa0-0e3f9fd11ab5-kube-api-access-hh9dq\") pod \"ingress-canary-vbsgm\" (UID: \"2aceb1e5-bf1c-4344-aaa0-0e3f9fd11ab5\") " pod="openshift-ingress-canary/ingress-canary-vbsgm" Jul 18 02:40:11.394548 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.394332 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b893a4ff-cd96-48b6-958f-a7ac7b736143-installation-pull-secrets\") pod \"image-registry-6b7d8ff6c9-qz7qp\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:40:11.394548 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.394356 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5qt5\" (UniqueName: \"kubernetes.io/projected/2f3ca073-4807-44e8-9e0e-1b9b8850bd6f-kube-api-access-v5qt5\") pod \"managed-serviceaccount-addon-agent-58fdcb5c84-jwllm\" (UID: \"2f3ca073-4807-44e8-9e0e-1b9b8850bd6f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58fdcb5c84-jwllm" Jul 18 02:40:11.394548 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.394378 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e95a777-c662-4a6a-bac1-9fa079e5914c-metrics-tls\") pod \"dns-default-w4wvr\" (UID: \"7e95a777-c662-4a6a-bac1-9fa079e5914c\") " pod="openshift-dns/dns-default-w4wvr" Jul 18 02:40:11.394548 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.394415 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b893a4ff-cd96-48b6-958f-a7ac7b736143-trusted-ca\") pod \"image-registry-6b7d8ff6c9-qz7qp\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:40:11.394548 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.394436 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5fa2fe3a-5e31-47f9-980e-ee1ee96925d9-klusterlet-config\") pod \"klusterlet-addon-workmgr-695bfc9c5-tgcx7\" (UID: \"5fa2fe3a-5e31-47f9-980e-ee1ee96925d9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695bfc9c5-tgcx7" Jul 18 02:40:11.394548 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.394467 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-tls\") pod \"image-registry-6b7d8ff6c9-qz7qp\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:40:11.394548 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.394488 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-bound-sa-token\") pod \"image-registry-6b7d8ff6c9-qz7qp\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:40:11.394548 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.394514 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnsh4\" (UniqueName: \"kubernetes.io/projected/5fa2fe3a-5e31-47f9-980e-ee1ee96925d9-kube-api-access-gnsh4\") pod \"klusterlet-addon-workmgr-695bfc9c5-tgcx7\" (UID: \"5fa2fe3a-5e31-47f9-980e-ee1ee96925d9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695bfc9c5-tgcx7" Jul 18 02:40:11.394548 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.394537 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2f3ca073-4807-44e8-9e0e-1b9b8850bd6f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-58fdcb5c84-jwllm\" (UID: \"2f3ca073-4807-44e8-9e0e-1b9b8850bd6f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58fdcb5c84-jwllm" Jul 18 02:40:11.395009 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.394561 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2aceb1e5-bf1c-4344-aaa0-0e3f9fd11ab5-cert\") pod \"ingress-canary-vbsgm\" (UID: \"2aceb1e5-bf1c-4344-aaa0-0e3f9fd11ab5\") " pod="openshift-ingress-canary/ingress-canary-vbsgm" Jul 18 02:40:11.395009 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.394590 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx299\" (UniqueName: \"kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-kube-api-access-dx299\") pod \"image-registry-6b7d8ff6c9-qz7qp\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:40:11.395009 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.394616 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e95a777-c662-4a6a-bac1-9fa079e5914c-tmp-dir\") pod \"dns-default-w4wvr\" (UID: \"7e95a777-c662-4a6a-bac1-9fa079e5914c\") " pod="openshift-dns/dns-default-w4wvr" Jul 18 02:40:11.395009 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.394698 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fa2fe3a-5e31-47f9-980e-ee1ee96925d9-tmp\") pod \"klusterlet-addon-workmgr-695bfc9c5-tgcx7\" (UID: \"5fa2fe3a-5e31-47f9-980e-ee1ee96925d9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695bfc9c5-tgcx7" Jul 18 02:40:11.395009 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.394705 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b893a4ff-cd96-48b6-958f-a7ac7b736143-ca-trust-extracted\") pod \"image-registry-6b7d8ff6c9-qz7qp\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:40:11.395009 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.394821 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7e95a777-c662-4a6a-bac1-9fa079e5914c-config-volume\") pod \"dns-default-w4wvr\" (UID: \"7e95a777-c662-4a6a-bac1-9fa079e5914c\") " pod="openshift-dns/dns-default-w4wvr" Jul 18 02:40:11.395009 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.394843 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-certificates\") pod \"image-registry-6b7d8ff6c9-qz7qp\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:40:11.395009 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.394902 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e95a777-c662-4a6a-bac1-9fa079e5914c-tmp-dir\") pod \"dns-default-w4wvr\" (UID: \"7e95a777-c662-4a6a-bac1-9fa079e5914c\") " pod="openshift-dns/dns-default-w4wvr" Jul 18 02:40:11.395009 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:11.394942 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 18 02:40:11.395009 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:11.394957 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp: secret "image-registry-tls" not found Jul 18 02:40:11.395009 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:11.395017 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-tls podName:b893a4ff-cd96-48b6-958f-a7ac7b736143 nodeName:}" failed. No retries permitted until 2026-07-18 02:40:11.89500049 +0000 UTC m=+35.498648158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-tls") pod "image-registry-6b7d8ff6c9-qz7qp" (UID: "b893a4ff-cd96-48b6-958f-a7ac7b736143") : secret "image-registry-tls" not found Jul 18 02:40:11.395830 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:11.395144 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 02:40:11.395830 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:11.395214 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e95a777-c662-4a6a-bac1-9fa079e5914c-metrics-tls podName:7e95a777-c662-4a6a-bac1-9fa079e5914c nodeName:}" failed. No retries permitted until 2026-07-18 02:40:11.895175066 +0000 UTC m=+35.498822744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e95a777-c662-4a6a-bac1-9fa079e5914c-metrics-tls") pod "dns-default-w4wvr" (UID: "7e95a777-c662-4a6a-bac1-9fa079e5914c") : secret "dns-default-metrics-tls" not found Jul 18 02:40:11.395830 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.395822 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b893a4ff-cd96-48b6-958f-a7ac7b736143-trusted-ca\") pod \"image-registry-6b7d8ff6c9-qz7qp\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:40:11.396934 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.396890 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b893a4ff-cd96-48b6-958f-a7ac7b736143-image-registry-private-configuration\") pod \"image-registry-6b7d8ff6c9-qz7qp\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:40:11.397216 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.397178 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2aceb1e5-bf1c-4344-aaa0-0e3f9fd11ab5-cert\") pod \"ingress-canary-vbsgm\" (UID: \"2aceb1e5-bf1c-4344-aaa0-0e3f9fd11ab5\") " pod="openshift-ingress-canary/ingress-canary-vbsgm" Jul 18 02:40:11.397361 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.397337 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b893a4ff-cd96-48b6-958f-a7ac7b736143-installation-pull-secrets\") pod \"image-registry-6b7d8ff6c9-qz7qp\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:40:11.397568 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.397551 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5fa2fe3a-5e31-47f9-980e-ee1ee96925d9-klusterlet-config\") pod \"klusterlet-addon-workmgr-695bfc9c5-tgcx7\" (UID: \"5fa2fe3a-5e31-47f9-980e-ee1ee96925d9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695bfc9c5-tgcx7" Jul 18 02:40:11.397663 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.397644 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2f3ca073-4807-44e8-9e0e-1b9b8850bd6f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-58fdcb5c84-jwllm\" (UID: \"2f3ca073-4807-44e8-9e0e-1b9b8850bd6f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58fdcb5c84-jwllm" Jul 18 02:40:11.401779 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.401753 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh9dq\" (UniqueName: \"kubernetes.io/projected/2aceb1e5-bf1c-4344-aaa0-0e3f9fd11ab5-kube-api-access-hh9dq\") pod \"ingress-canary-vbsgm\" (UID: \"2aceb1e5-bf1c-4344-aaa0-0e3f9fd11ab5\") " pod="openshift-ingress-canary/ingress-canary-vbsgm" Jul 18 02:40:11.405372 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.405347 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9bb7\" (UniqueName: \"kubernetes.io/projected/7e95a777-c662-4a6a-bac1-9fa079e5914c-kube-api-access-z9bb7\") pod \"dns-default-w4wvr\" (UID: \"7e95a777-c662-4a6a-bac1-9fa079e5914c\") " pod="openshift-dns/dns-default-w4wvr" Jul 18 02:40:11.406443 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.406421 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx299\" (UniqueName: \"kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-kube-api-access-dx299\") pod \"image-registry-6b7d8ff6c9-qz7qp\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:40:11.406443 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.406436 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-bound-sa-token\") pod \"image-registry-6b7d8ff6c9-qz7qp\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:40:11.406553 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.406479 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5qt5\" (UniqueName: \"kubernetes.io/projected/2f3ca073-4807-44e8-9e0e-1b9b8850bd6f-kube-api-access-v5qt5\") pod \"managed-serviceaccount-addon-agent-58fdcb5c84-jwllm\" (UID: \"2f3ca073-4807-44e8-9e0e-1b9b8850bd6f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58fdcb5c84-jwllm" Jul 18 02:40:11.406587 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.406556 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnsh4\" (UniqueName: \"kubernetes.io/projected/5fa2fe3a-5e31-47f9-980e-ee1ee96925d9-kube-api-access-gnsh4\") pod \"klusterlet-addon-workmgr-695bfc9c5-tgcx7\" (UID: \"5fa2fe3a-5e31-47f9-980e-ee1ee96925d9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695bfc9c5-tgcx7" Jul 18 02:40:11.507580 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.507552 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" Jul 18 02:40:11.516926 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.516895 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695bfc9c5-tgcx7" Jul 18 02:40:11.529579 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.529559 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58fdcb5c84-jwllm" Jul 18 02:40:11.551226 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.551196 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vbsgm" Jul 18 02:40:11.684229 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.684094 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-695bfc9c5-tgcx7"] Jul 18 02:40:11.702645 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.702595 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58fdcb5c84-jwllm"] Jul 18 02:40:11.706667 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:40:11.706490 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f3ca073_4807_44e8_9e0e_1b9b8850bd6f.slice/crio-ec24a56a92b72b772578acba014190bb8d00f9171f38d1d052d955c90a9dd074 WatchSource:0}: Error finding container ec24a56a92b72b772578acba014190bb8d00f9171f38d1d052d955c90a9dd074: Status 404 returned error can't find the container with id ec24a56a92b72b772578acba014190bb8d00f9171f38d1d052d955c90a9dd074 Jul 18 02:40:11.714567 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.714526 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8"] Jul 18 02:40:11.717210 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:40:11.717162 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0825f880_ea1c_4683_b9b1_2e9aaeee31d8.slice/crio-581c9f375ccf96c547d80d8fb27278d58bfb531e45adbfa6b2559091ba3ba380 WatchSource:0}: Error finding container 581c9f375ccf96c547d80d8fb27278d58bfb531e45adbfa6b2559091ba3ba380: Status 404 returned error can't find the container with id 581c9f375ccf96c547d80d8fb27278d58bfb531e45adbfa6b2559091ba3ba380 Jul 18 02:40:11.739935 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.739914 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vbsgm"] Jul 18 02:40:11.742454 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:40:11.742434 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aceb1e5_bf1c_4344_aaa0_0e3f9fd11ab5.slice/crio-d43639649ce1f1888100ec7fa77997d2027461a188c87bc422e63829a6eafb3c WatchSource:0}: Error finding container d43639649ce1f1888100ec7fa77997d2027461a188c87bc422e63829a6eafb3c: Status 404 returned error can't find the container with id d43639649ce1f1888100ec7fa77997d2027461a188c87bc422e63829a6eafb3c Jul 18 02:40:11.898371 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.898341 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e95a777-c662-4a6a-bac1-9fa079e5914c-metrics-tls\") pod \"dns-default-w4wvr\" (UID: \"7e95a777-c662-4a6a-bac1-9fa079e5914c\") " pod="openshift-dns/dns-default-w4wvr" Jul 18 02:40:11.898514 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:11.898400 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-tls\") pod \"image-registry-6b7d8ff6c9-qz7qp\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:40:11.898514 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:11.898503 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 02:40:11.898614 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:11.898569 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 18 02:40:11.898614 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:11.898590 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp: secret "image-registry-tls" not found Jul 18 02:40:11.898699 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:11.898576 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e95a777-c662-4a6a-bac1-9fa079e5914c-metrics-tls podName:7e95a777-c662-4a6a-bac1-9fa079e5914c nodeName:}" failed. No retries permitted until 2026-07-18 02:40:12.89855583 +0000 UTC m=+36.502203507 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e95a777-c662-4a6a-bac1-9fa079e5914c-metrics-tls") pod "dns-default-w4wvr" (UID: "7e95a777-c662-4a6a-bac1-9fa079e5914c") : secret "dns-default-metrics-tls" not found Jul 18 02:40:11.898699 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:11.898662 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-tls podName:b893a4ff-cd96-48b6-958f-a7ac7b736143 nodeName:}" failed. No retries permitted until 2026-07-18 02:40:12.898643327 +0000 UTC m=+36.502291004 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-tls") pod "image-registry-6b7d8ff6c9-qz7qp" (UID: "b893a4ff-cd96-48b6-958f-a7ac7b736143") : secret "image-registry-tls" not found Jul 18 02:40:12.159437 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:12.159408 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" event={"ID":"0825f880-ea1c-4683-b9b1-2e9aaeee31d8","Type":"ContainerStarted","Data":"581c9f375ccf96c547d80d8fb27278d58bfb531e45adbfa6b2559091ba3ba380"} Jul 18 02:40:12.160434 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:12.160412 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vbsgm" event={"ID":"2aceb1e5-bf1c-4344-aaa0-0e3f9fd11ab5","Type":"ContainerStarted","Data":"d43639649ce1f1888100ec7fa77997d2027461a188c87bc422e63829a6eafb3c"} Jul 18 02:40:12.161398 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:12.161369 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58fdcb5c84-jwllm" event={"ID":"2f3ca073-4807-44e8-9e0e-1b9b8850bd6f","Type":"ContainerStarted","Data":"ec24a56a92b72b772578acba014190bb8d00f9171f38d1d052d955c90a9dd074"} Jul 18 02:40:12.162332 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:12.162312 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695bfc9c5-tgcx7" event={"ID":"5fa2fe3a-5e31-47f9-980e-ee1ee96925d9","Type":"ContainerStarted","Data":"74c35a90ab3e355e725182a6b261374b79be3a365dbb59aee24c64c4d295949d"} Jul 18 02:40:12.894627 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:12.894592 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:40:12.895092 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:12.895076 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:40:12.895560 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:12.895541 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:40:12.899246 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:12.899227 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 18 02:40:12.899813 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:12.899716 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 18 02:40:12.899980 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:12.899961 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zt6zk\"" Jul 18 02:40:12.900177 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:12.900161 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 18 02:40:12.900485 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:12.900469 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 18 02:40:12.900681 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:12.900658 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hzcn2\"" Jul 18 02:40:12.905607 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:12.905567 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e95a777-c662-4a6a-bac1-9fa079e5914c-metrics-tls\") pod \"dns-default-w4wvr\" (UID: \"7e95a777-c662-4a6a-bac1-9fa079e5914c\") " pod="openshift-dns/dns-default-w4wvr" Jul 18 02:40:12.905755 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:12.905619 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-tls\") pod \"image-registry-6b7d8ff6c9-qz7qp\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:40:12.905815 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:12.905760 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 18 02:40:12.905815 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:12.905773 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp: secret "image-registry-tls" not found Jul 18 02:40:12.905914 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:12.905823 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-tls podName:b893a4ff-cd96-48b6-958f-a7ac7b736143 nodeName:}" failed. No retries permitted until 2026-07-18 02:40:14.905806106 +0000 UTC m=+38.509453790 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-tls") pod "image-registry-6b7d8ff6c9-qz7qp" (UID: "b893a4ff-cd96-48b6-958f-a7ac7b736143") : secret "image-registry-tls" not found Jul 18 02:40:12.906207 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:12.906152 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 02:40:12.906315 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:12.906218 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e95a777-c662-4a6a-bac1-9fa079e5914c-metrics-tls podName:7e95a777-c662-4a6a-bac1-9fa079e5914c nodeName:}" failed. No retries permitted until 2026-07-18 02:40:14.906203543 +0000 UTC m=+38.509851207 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e95a777-c662-4a6a-bac1-9fa079e5914c-metrics-tls") pod "dns-default-w4wvr" (UID: "7e95a777-c662-4a6a-bac1-9fa079e5914c") : secret "dns-default-metrics-tls" not found Jul 18 02:40:13.802502 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:13.802270 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6v9b_76644748-24e4-4b1d-a6b1-68887436724c/konnectivity-agent/0.log" limit=1024 Jul 18 02:40:13.926950 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:13.926889 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6v9b_76644748-24e4-4b1d-a6b1-68887436724c/konnectivity-agent/0.log" limit=1024 Jul 18 02:40:14.027446 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:14.027367 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6v9b_76644748-24e4-4b1d-a6b1-68887436724c/konnectivity-agent/0.log" limit=1024 Jul 18 02:40:14.926966 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:14.926925 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e95a777-c662-4a6a-bac1-9fa079e5914c-metrics-tls\") pod \"dns-default-w4wvr\" (UID: \"7e95a777-c662-4a6a-bac1-9fa079e5914c\") " pod="openshift-dns/dns-default-w4wvr" Jul 18 02:40:14.927498 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:14.926999 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-tls\") pod \"image-registry-6b7d8ff6c9-qz7qp\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:40:14.927498 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:14.927122 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 02:40:14.927498 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:14.927177 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 18 02:40:14.927498 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:14.927207 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp: secret "image-registry-tls" not found Jul 18 02:40:14.927498 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:14.927213 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e95a777-c662-4a6a-bac1-9fa079e5914c-metrics-tls podName:7e95a777-c662-4a6a-bac1-9fa079e5914c nodeName:}" failed. No retries permitted until 2026-07-18 02:40:18.927178227 +0000 UTC m=+42.530825891 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e95a777-c662-4a6a-bac1-9fa079e5914c-metrics-tls") pod "dns-default-w4wvr" (UID: "7e95a777-c662-4a6a-bac1-9fa079e5914c") : secret "dns-default-metrics-tls" not found Jul 18 02:40:14.927498 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:14.927255 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-tls podName:b893a4ff-cd96-48b6-958f-a7ac7b736143 nodeName:}" failed. No retries permitted until 2026-07-18 02:40:18.927240328 +0000 UTC m=+42.530888019 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-tls") pod "image-registry-6b7d8ff6c9-qz7qp" (UID: "b893a4ff-cd96-48b6-958f-a7ac7b736143") : secret "image-registry-tls" not found Jul 18 02:40:16.349136 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:16.349100 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6v9b_76644748-24e4-4b1d-a6b1-68887436724c/konnectivity-agent/0.log" limit=1024 Jul 18 02:40:16.402994 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:16.402956 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6v9b_76644748-24e4-4b1d-a6b1-68887436724c/konnectivity-agent/0.log" limit=1024 Jul 18 02:40:18.960622 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:18.960592 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e95a777-c662-4a6a-bac1-9fa079e5914c-metrics-tls\") pod \"dns-default-w4wvr\" (UID: \"7e95a777-c662-4a6a-bac1-9fa079e5914c\") " pod="openshift-dns/dns-default-w4wvr" Jul 18 02:40:18.961017 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:18.960655 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-tls\") pod \"image-registry-6b7d8ff6c9-qz7qp\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:40:18.961017 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:18.960784 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 02:40:18.961017 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:18.960829 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 18 02:40:18.961017 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:18.960844 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp: secret "image-registry-tls" not found Jul 18 02:40:18.961017 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:18.960850 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e95a777-c662-4a6a-bac1-9fa079e5914c-metrics-tls podName:7e95a777-c662-4a6a-bac1-9fa079e5914c nodeName:}" failed. No retries permitted until 2026-07-18 02:40:26.9608307 +0000 UTC m=+50.564478365 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e95a777-c662-4a6a-bac1-9fa079e5914c-metrics-tls") pod "dns-default-w4wvr" (UID: "7e95a777-c662-4a6a-bac1-9fa079e5914c") : secret "dns-default-metrics-tls" not found Jul 18 02:40:18.961017 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:18.960895 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-tls podName:b893a4ff-cd96-48b6-958f-a7ac7b736143 nodeName:}" failed. No retries permitted until 2026-07-18 02:40:26.960882754 +0000 UTC m=+50.564530418 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-tls") pod "image-registry-6b7d8ff6c9-qz7qp" (UID: "b893a4ff-cd96-48b6-958f-a7ac7b736143") : secret "image-registry-tls" not found Jul 18 02:40:21.189329 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:21.189300 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" event={"ID":"0825f880-ea1c-4683-b9b1-2e9aaeee31d8","Type":"ContainerStarted","Data":"1ab81d2ed2bc9cdbad3c73228a0cc16cb2d26251e15498528dad0e84f02b8cf0"} Jul 18 02:40:22.193963 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:22.193930 2585 generic.go:356] "Generic (PLEG): container finished" podID="35cb46c9-0c8a-4562-a059-521ad01f2552" containerID="1b7d4b31a284f12890e370e5a240b0b93d66cfea649c25f5ec88faa7f7441d25" exitCode=0 Jul 18 02:40:22.194325 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:22.194011 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvpps" event={"ID":"35cb46c9-0c8a-4562-a059-521ad01f2552","Type":"ContainerDied","Data":"1b7d4b31a284f12890e370e5a240b0b93d66cfea649c25f5ec88faa7f7441d25"} Jul 18 02:40:22.195982 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:22.195956 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" event={"ID":"0825f880-ea1c-4683-b9b1-2e9aaeee31d8","Type":"ContainerStarted","Data":"bd137bcb93a9d988a172dee503af082e8b027209927135ca1106e6c48f5297d2"} Jul 18 02:40:22.196067 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:22.195995 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" event={"ID":"0825f880-ea1c-4683-b9b1-2e9aaeee31d8","Type":"ContainerStarted","Data":"41c8d89a0a6a05886788f9b998916057f21b668d2613bd86690f890b20093aa7"} Jul 18 02:40:22.197341 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:22.197322 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vbsgm" event={"ID":"2aceb1e5-bf1c-4344-aaa0-0e3f9fd11ab5","Type":"ContainerStarted","Data":"1291702b89ae269820c09d3a33c2dbd295f198b0466371b75e1f93cf2a02b62b"} Jul 18 02:40:22.198661 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:22.198639 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58fdcb5c84-jwllm" event={"ID":"2f3ca073-4807-44e8-9e0e-1b9b8850bd6f","Type":"ContainerStarted","Data":"442e9e06399e0cd72c94ef1583e1bedc33c8348c0551b2446cc6397083b04016"} Jul 18 02:40:22.199936 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:22.199914 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695bfc9c5-tgcx7" event={"ID":"5fa2fe3a-5e31-47f9-980e-ee1ee96925d9","Type":"ContainerStarted","Data":"6e0707e95fb86b418f5520ec474982a200df2376b961879caedff6ffc8447e1d"} Jul 18 02:40:22.200096 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:22.200085 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695bfc9c5-tgcx7" Jul 18 02:40:22.201898 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:22.201854 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695bfc9c5-tgcx7" Jul 18 02:40:22.229717 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:22.229672 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695bfc9c5-tgcx7" podStartSLOduration=15.897792136 podStartE2EDuration="25.229658268s" totalImagesPullingTime="9.331866132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:39:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:40:22.229406461 +0000 UTC m=+45.833054145" watchObservedRunningTime="2026-07-18 02:40:22.229658268 +0000 UTC m=+45.833305933" Jul 18 02:40:22.243595 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:22.243549 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vbsgm" podStartSLOduration=1.9903735980000001 podStartE2EDuration="11.24353924s" totalImagesPullingTime="9.253165642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:40:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:40:22.24326748 +0000 UTC m=+45.846915166" watchObservedRunningTime="2026-07-18 02:40:22.24353924 +0000 UTC m=+45.847186927" Jul 18 02:40:22.261534 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:22.261491 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" podStartSLOduration=15.98324095 podStartE2EDuration="25.261476374s" totalImagesPullingTime="9.278235424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:39:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:40:22.261143866 +0000 UTC m=+45.864791552" watchObservedRunningTime="2026-07-18 02:40:22.261476374 +0000 UTC m=+45.865124061" Jul 18 02:40:22.275208 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:22.275158 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58fdcb5c84-jwllm" podStartSLOduration=15.94794512 podStartE2EDuration="25.275147558s" totalImagesPullingTime="9.327202438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:39:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:40:22.275021603 +0000 UTC m=+45.878669290" watchObservedRunningTime="2026-07-18 02:40:22.275147558 +0000 UTC m=+45.878795242" Jul 18 02:40:23.204547 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:23.204514 2585 generic.go:356] "Generic (PLEG): container finished" podID="35cb46c9-0c8a-4562-a059-521ad01f2552" containerID="45fcd3494520e53eac0344167c34bf9914894a2e8c19d9dfbac9d30b8adaf29f" exitCode=0 Jul 18 02:40:23.204921 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:23.204612 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvpps" event={"ID":"35cb46c9-0c8a-4562-a059-521ad01f2552","Type":"ContainerDied","Data":"45fcd3494520e53eac0344167c34bf9914894a2e8c19d9dfbac9d30b8adaf29f"} Jul 18 02:40:24.210293 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:24.210259 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvpps" event={"ID":"35cb46c9-0c8a-4562-a059-521ad01f2552","Type":"ContainerStarted","Data":"0e511540b96e1aa12748c62e1319a8c51227129318b5c519ae4ee40e59388394"} Jul 18 02:40:24.232909 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:24.232822 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xvpps" podStartSLOduration=16.735020798 podStartE2EDuration="47.232804026s" totalImagesPullingTime="30.497783228s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 02:39:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:40:24.231301484 +0000 UTC m=+47.834949166" watchObservedRunningTime="2026-07-18 02:40:24.232804026 +0000 UTC m=+47.836451712" Jul 18 02:40:24.351834 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:24.351804 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6v9b_76644748-24e4-4b1d-a6b1-68887436724c/konnectivity-agent/0.log" limit=1024 Jul 18 02:40:24.404803 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:24.404772 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h6v9b_76644748-24e4-4b1d-a6b1-68887436724c/konnectivity-agent/0.log" limit=1024 Jul 18 02:40:27.028974 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:27.028938 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e95a777-c662-4a6a-bac1-9fa079e5914c-metrics-tls\") pod \"dns-default-w4wvr\" (UID: \"7e95a777-c662-4a6a-bac1-9fa079e5914c\") " pod="openshift-dns/dns-default-w4wvr" Jul 18 02:40:27.029347 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:27.028986 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-tls\") pod \"image-registry-6b7d8ff6c9-qz7qp\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:40:27.029347 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:27.029068 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 18 02:40:27.029347 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:27.029069 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 02:40:27.029347 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:27.029120 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e95a777-c662-4a6a-bac1-9fa079e5914c-metrics-tls podName:7e95a777-c662-4a6a-bac1-9fa079e5914c nodeName:}" failed. No retries permitted until 2026-07-18 02:40:43.029106739 +0000 UTC m=+66.632754403 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e95a777-c662-4a6a-bac1-9fa079e5914c-metrics-tls") pod "dns-default-w4wvr" (UID: "7e95a777-c662-4a6a-bac1-9fa079e5914c") : secret "dns-default-metrics-tls" not found Jul 18 02:40:27.029347 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:27.029079 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp: secret "image-registry-tls" not found Jul 18 02:40:27.029347 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:27.029211 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-tls podName:b893a4ff-cd96-48b6-958f-a7ac7b736143 nodeName:}" failed. No retries permitted until 2026-07-18 02:40:43.029196719 +0000 UTC m=+66.632844397 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-tls") pod "image-registry-6b7d8ff6c9-qz7qp" (UID: "b893a4ff-cd96-48b6-958f-a7ac7b736143") : secret "image-registry-tls" not found Jul 18 02:40:33.625032 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.624998 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-gchp4"] Jul 18 02:40:33.629296 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.629276 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-gchp4" Jul 18 02:40:33.632117 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.632100 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 18 02:40:33.633268 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.633247 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-6g2fx\"" Jul 18 02:40:33.633268 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.633257 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 18 02:40:33.633422 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.633278 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 18 02:40:33.633422 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.633288 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 18 02:40:33.641552 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.641532 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-scxbj"] Jul 18 02:40:33.645543 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.645523 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hjrpw"] Jul 18 02:40:33.645640 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.645630 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-scxbj" Jul 18 02:40:33.648338 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.648319 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-6dhq7\"" Jul 18 02:40:33.648338 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.648333 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 18 02:40:33.648471 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.648322 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 18 02:40:33.649592 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.649572 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-krtsg"] Jul 18 02:40:33.649694 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.649681 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hjrpw" Jul 18 02:40:33.652403 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.652387 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 18 02:40:33.652496 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.652480 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 18 02:40:33.652556 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.652506 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 18 02:40:33.653044 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.653025 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-jb8kd\"" Jul 18 02:40:33.653412 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.653395 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-gchp4"] Jul 18 02:40:33.653494 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.653481 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-krtsg" Jul 18 02:40:33.655824 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.655803 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hjrpw"] Jul 18 02:40:33.658746 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.658728 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 18 02:40:33.659053 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.659019 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 18 02:40:33.659216 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.658832 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 18 02:40:33.659473 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.659288 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-hk5wj\"" Jul 18 02:40:33.659776 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.659759 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 18 02:40:33.660951 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.660927 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-krtsg"] Jul 18 02:40:33.661877 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.661838 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-scxbj"] Jul 18 02:40:33.665364 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.665341 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 18 02:40:33.725605 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.725587 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-27nmz"] Jul 18 02:40:33.729285 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.729269 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-c4l7k"] Jul 18 02:40:33.729392 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.729378 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-27nmz" Jul 18 02:40:33.731693 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.731678 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-s2rz2\"" Jul 18 02:40:33.732907 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.732883 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-c4l7k" Jul 18 02:40:33.733573 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.733556 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6l557"] Jul 18 02:40:33.735053 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.735033 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 18 02:40:33.735270 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.735249 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 18 02:40:33.735379 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.735322 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mblvg\"" Jul 18 02:40:33.735379 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.735373 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 18 02:40:33.735705 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.735684 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 18 02:40:33.737467 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.737446 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-d5t6p"] Jul 18 02:40:33.737562 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.737546 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6l557" Jul 18 02:40:33.739862 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.739841 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 18 02:40:33.740125 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.740109 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 18 02:40:33.741294 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.741274 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 18 02:40:33.741844 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.741595 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-789b768784-4p68t"] Jul 18 02:40:33.742406 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.742385 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-d5t6p" Jul 18 02:40:33.744909 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.744878 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xgl4x\"" Jul 18 02:40:33.745984 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.745964 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 18 02:40:33.746775 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.746756 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 18 02:40:33.746858 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.746812 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-twr6t\"" Jul 18 02:40:33.747196 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.747168 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 18 02:40:33.747523 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.747507 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 18 02:40:33.747868 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.747849 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 18 02:40:33.749001 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.748974 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-27nmz"] Jul 18 02:40:33.749112 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.749009 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-c4l7k"] Jul 18 02:40:33.749112 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.749079 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:40:33.749982 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.749962 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6l557"] Jul 18 02:40:33.752709 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.752690 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 18 02:40:33.754087 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.754043 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 18 02:40:33.754087 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.754099 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8lwll\"" Jul 18 02:40:33.754087 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.754107 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 18 02:40:33.754087 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.754161 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 18 02:40:33.754087 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.754052 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 18 02:40:33.754708 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.754350 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 18 02:40:33.754708 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.754392 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 18 02:40:33.756250 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.756222 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-d5t6p"] Jul 18 02:40:33.758697 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.758645 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-789b768784-4p68t"] Jul 18 02:40:33.773573 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.773554 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fadc8c8-588f-41be-94e8-4c7de1ce8649-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-hjrpw\" (UID: \"6fadc8c8-588f-41be-94e8-4c7de1ce8649\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hjrpw" Jul 18 02:40:33.773644 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.773585 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/079022d6-f624-49c4-93a6-7bdb4b6a9ffd-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-scxbj\" (UID: \"079022d6-f624-49c4-93a6-7bdb4b6a9ffd\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-scxbj" Jul 18 02:40:33.773644 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.773606 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzzpx\" (UniqueName: \"kubernetes.io/projected/709575dc-6eae-43e1-9208-cc228795929a-kube-api-access-bzzpx\") pod \"console-operator-565b7b569d-krtsg\" (UID: \"709575dc-6eae-43e1-9208-cc228795929a\") " pod="openshift-console-operator/console-operator-565b7b569d-krtsg" Jul 18 02:40:33.773710 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.773687 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/079022d6-f624-49c4-93a6-7bdb4b6a9ffd-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-scxbj\" (UID: \"079022d6-f624-49c4-93a6-7bdb4b6a9ffd\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-scxbj" Jul 18 02:40:33.773744 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.773710 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcfsw\" (UniqueName: \"kubernetes.io/projected/6fadc8c8-588f-41be-94e8-4c7de1ce8649-kube-api-access-bcfsw\") pod \"cluster-samples-operator-79b7c55f87-hjrpw\" (UID: \"6fadc8c8-588f-41be-94e8-4c7de1ce8649\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hjrpw" Jul 18 02:40:33.773744 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.773730 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7g52\" (UniqueName: \"kubernetes.io/projected/53d15949-a9fb-4897-bec5-feb443eb99d9-kube-api-access-h7g52\") pod \"service-ca-operator-5bcbc94c78-gchp4\" (UID: \"53d15949-a9fb-4897-bec5-feb443eb99d9\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-gchp4" Jul 18 02:40:33.773800 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.773747 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/709575dc-6eae-43e1-9208-cc228795929a-config\") pod \"console-operator-565b7b569d-krtsg\" (UID: \"709575dc-6eae-43e1-9208-cc228795929a\") " pod="openshift-console-operator/console-operator-565b7b569d-krtsg" Jul 18 02:40:33.773800 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.773785 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/53d15949-a9fb-4897-bec5-feb443eb99d9-config\") pod \"service-ca-operator-5bcbc94c78-gchp4\" (UID: \"53d15949-a9fb-4897-bec5-feb443eb99d9\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-gchp4" Jul 18 02:40:33.773855 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.773812 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/709575dc-6eae-43e1-9208-cc228795929a-serving-cert\") pod \"console-operator-565b7b569d-krtsg\" (UID: \"709575dc-6eae-43e1-9208-cc228795929a\") " pod="openshift-console-operator/console-operator-565b7b569d-krtsg" Jul 18 02:40:33.773855 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.773832 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/53d15949-a9fb-4897-bec5-feb443eb99d9-serving-cert\") pod \"service-ca-operator-5bcbc94c78-gchp4\" (UID: \"53d15949-a9fb-4897-bec5-feb443eb99d9\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-gchp4" Jul 18 02:40:33.773917 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.773857 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/709575dc-6eae-43e1-9208-cc228795929a-trusted-ca\") pod \"console-operator-565b7b569d-krtsg\" (UID: \"709575dc-6eae-43e1-9208-cc228795929a\") " pod="openshift-console-operator/console-operator-565b7b569d-krtsg" Jul 18 02:40:33.828039 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.828016 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-44wcx"] Jul 18 02:40:33.831431 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.831418 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-44wcx" Jul 18 02:40:33.833916 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.833897 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 18 02:40:33.834032 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.834017 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 18 02:40:33.834088 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.834017 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-82bmq\"" Jul 18 02:40:33.839573 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.839548 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-44wcx"] Jul 18 02:40:33.874965 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.874944 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52dc3a30-b4eb-40ce-8380-9fc3a18db5df-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-d5t6p\" (UID: \"52dc3a30-b4eb-40ce-8380-9fc3a18db5df\") " pod="openshift-insights/insights-operator-555d9d7885-d5t6p" Jul 18 02:40:33.875094 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.874972 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcfsw\" (UniqueName: \"kubernetes.io/projected/6fadc8c8-588f-41be-94e8-4c7de1ce8649-kube-api-access-bcfsw\") pod \"cluster-samples-operator-79b7c55f87-hjrpw\" (UID: \"6fadc8c8-588f-41be-94e8-4c7de1ce8649\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hjrpw" Jul 18 02:40:33.875094 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.874990 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7g52\" (UniqueName: \"kubernetes.io/projected/53d15949-a9fb-4897-bec5-feb443eb99d9-kube-api-access-h7g52\") pod \"service-ca-operator-5bcbc94c78-gchp4\" (UID: \"53d15949-a9fb-4897-bec5-feb443eb99d9\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-gchp4" Jul 18 02:40:33.875094 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.875010 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/709575dc-6eae-43e1-9208-cc228795929a-config\") pod \"console-operator-565b7b569d-krtsg\" (UID: \"709575dc-6eae-43e1-9208-cc228795929a\") " pod="openshift-console-operator/console-operator-565b7b569d-krtsg" Jul 18 02:40:33.875094 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.875065 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-metrics-certs\") pod \"router-default-789b768784-4p68t\" (UID: \"7ce9775c-ae6f-4891-b9df-58d8a04ec1c8\") " pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:40:33.875327 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.875121 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp4z6\" (UniqueName: \"kubernetes.io/projected/8353a5f3-694a-40ac-bde3-a22bc16e47b0-kube-api-access-xp4z6\") pod \"network-check-source-7bc5b6bf85-27nmz\" (UID: \"8353a5f3-694a-40ac-bde3-a22bc16e47b0\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-27nmz" Jul 18 02:40:33.875327 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.875163 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/53d15949-a9fb-4897-bec5-feb443eb99d9-config\") pod \"service-ca-operator-5bcbc94c78-gchp4\" (UID: \"53d15949-a9fb-4897-bec5-feb443eb99d9\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-gchp4" Jul 18 02:40:33.875327 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.875207 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-service-ca-bundle\") pod \"router-default-789b768784-4p68t\" (UID: \"7ce9775c-ae6f-4891-b9df-58d8a04ec1c8\") " pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:40:33.875327 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.875236 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-default-certificate\") pod \"router-default-789b768784-4p68t\" (UID: \"7ce9775c-ae6f-4891-b9df-58d8a04ec1c8\") " pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:40:33.875327 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.875268 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/709575dc-6eae-43e1-9208-cc228795929a-serving-cert\") pod \"console-operator-565b7b569d-krtsg\" (UID: \"709575dc-6eae-43e1-9208-cc228795929a\") " pod="openshift-console-operator/console-operator-565b7b569d-krtsg" Jul 18 02:40:33.875327 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.875294 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzzdl\" (UniqueName: \"kubernetes.io/projected/2f3a8a13-5e99-4e24-98ee-b112c8b3b784-kube-api-access-jzzdl\") pod \"cluster-monitoring-operator-76cf8966f8-6l557\" (UID: \"2f3a8a13-5e99-4e24-98ee-b112c8b3b784\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6l557" Jul 18 02:40:33.875327 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.875318 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/52dc3a30-b4eb-40ce-8380-9fc3a18db5df-snapshots\") pod \"insights-operator-555d9d7885-d5t6p\" (UID: \"52dc3a30-b4eb-40ce-8380-9fc3a18db5df\") " pod="openshift-insights/insights-operator-555d9d7885-d5t6p" Jul 18 02:40:33.875655 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.875342 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drh7b\" (UniqueName: \"kubernetes.io/projected/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-kube-api-access-drh7b\") pod \"router-default-789b768784-4p68t\" (UID: \"7ce9775c-ae6f-4891-b9df-58d8a04ec1c8\") " pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:40:33.875655 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.875370 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-stats-auth\") pod \"router-default-789b768784-4p68t\" (UID: \"7ce9775c-ae6f-4891-b9df-58d8a04ec1c8\") " pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:40:33.875655 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.875394 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/53d15949-a9fb-4897-bec5-feb443eb99d9-serving-cert\") pod \"service-ca-operator-5bcbc94c78-gchp4\" (UID: \"53d15949-a9fb-4897-bec5-feb443eb99d9\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-gchp4" Jul 18 02:40:33.875655 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.875482 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/709575dc-6eae-43e1-9208-cc228795929a-trusted-ca\") pod \"console-operator-565b7b569d-krtsg\" (UID: \"709575dc-6eae-43e1-9208-cc228795929a\") " pod="openshift-console-operator/console-operator-565b7b569d-krtsg" Jul 18 02:40:33.875655 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.875530 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e05161ba-27ca-4679-9439-bad0efa4500f-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-c4l7k\" (UID: \"e05161ba-27ca-4679-9439-bad0efa4500f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-c4l7k" Jul 18 02:40:33.875655 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.875558 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52dc3a30-b4eb-40ce-8380-9fc3a18db5df-service-ca-bundle\") pod \"insights-operator-555d9d7885-d5t6p\" (UID: \"52dc3a30-b4eb-40ce-8380-9fc3a18db5df\") " pod="openshift-insights/insights-operator-555d9d7885-d5t6p" Jul 18 02:40:33.875655 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.875599 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fadc8c8-588f-41be-94e8-4c7de1ce8649-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-hjrpw\" (UID: \"6fadc8c8-588f-41be-94e8-4c7de1ce8649\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hjrpw" Jul 18 02:40:33.875655 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.875622 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2f3a8a13-5e99-4e24-98ee-b112c8b3b784-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-6l557\" (UID: \"2f3a8a13-5e99-4e24-98ee-b112c8b3b784\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6l557" Jul 18 02:40:33.876008 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.875680 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e05161ba-27ca-4679-9439-bad0efa4500f-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-c4l7k\" (UID: \"e05161ba-27ca-4679-9439-bad0efa4500f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-c4l7k" Jul 18 02:40:33.876008 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.875715 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qt2t\" (UniqueName: \"kubernetes.io/projected/e05161ba-27ca-4679-9439-bad0efa4500f-kube-api-access-9qt2t\") pod \"kube-storage-version-migrator-operator-7658d9b749-c4l7k\" (UID: \"e05161ba-27ca-4679-9439-bad0efa4500f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-c4l7k" Jul 18 02:40:33.876008 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.875722 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/53d15949-a9fb-4897-bec5-feb443eb99d9-config\") pod \"service-ca-operator-5bcbc94c78-gchp4\" (UID: \"53d15949-a9fb-4897-bec5-feb443eb99d9\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-gchp4" Jul 18 02:40:33.876008 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.875722 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/709575dc-6eae-43e1-9208-cc228795929a-config\") pod \"console-operator-565b7b569d-krtsg\" (UID: \"709575dc-6eae-43e1-9208-cc228795929a\") " pod="openshift-console-operator/console-operator-565b7b569d-krtsg" Jul 18 02:40:33.876008 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:33.875690 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 18 02:40:33.876008 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.875763 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/079022d6-f624-49c4-93a6-7bdb4b6a9ffd-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-scxbj\" (UID: \"079022d6-f624-49c4-93a6-7bdb4b6a9ffd\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-scxbj" Jul 18 02:40:33.876008 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:33.875823 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fadc8c8-588f-41be-94e8-4c7de1ce8649-samples-operator-tls podName:6fadc8c8-588f-41be-94e8-4c7de1ce8649 nodeName:}" failed. No retries permitted until 2026-07-18 02:40:34.375805409 +0000 UTC m=+57.979453074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6fadc8c8-588f-41be-94e8-4c7de1ce8649-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-hjrpw" (UID: "6fadc8c8-588f-41be-94e8-4c7de1ce8649") : secret "samples-operator-tls" not found Jul 18 02:40:33.876008 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:33.875851 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 18 02:40:33.876008 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.875871 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/52dc3a30-b4eb-40ce-8380-9fc3a18db5df-serving-cert\") pod \"insights-operator-555d9d7885-d5t6p\" (UID: \"52dc3a30-b4eb-40ce-8380-9fc3a18db5df\") " pod="openshift-insights/insights-operator-555d9d7885-d5t6p" Jul 18 02:40:33.876008 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:33.875910 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/079022d6-f624-49c4-93a6-7bdb4b6a9ffd-networking-console-plugin-cert podName:079022d6-f624-49c4-93a6-7bdb4b6a9ffd nodeName:}" failed. No retries permitted until 2026-07-18 02:40:34.375890325 +0000 UTC m=+57.979538004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/079022d6-f624-49c4-93a6-7bdb4b6a9ffd-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-scxbj" (UID: "079022d6-f624-49c4-93a6-7bdb4b6a9ffd") : secret "networking-console-plugin-cert" not found Jul 18 02:40:33.876008 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.875957 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2r9p\" (UniqueName: \"kubernetes.io/projected/52dc3a30-b4eb-40ce-8380-9fc3a18db5df-kube-api-access-d2r9p\") pod \"insights-operator-555d9d7885-d5t6p\" (UID: \"52dc3a30-b4eb-40ce-8380-9fc3a18db5df\") " pod="openshift-insights/insights-operator-555d9d7885-d5t6p" Jul 18 02:40:33.876008 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.875992 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzzpx\" (UniqueName: \"kubernetes.io/projected/709575dc-6eae-43e1-9208-cc228795929a-kube-api-access-bzzpx\") pod \"console-operator-565b7b569d-krtsg\" (UID: \"709575dc-6eae-43e1-9208-cc228795929a\") " pod="openshift-console-operator/console-operator-565b7b569d-krtsg" Jul 18 02:40:33.876515 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.876016 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f3a8a13-5e99-4e24-98ee-b112c8b3b784-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-6l557\" (UID: \"2f3a8a13-5e99-4e24-98ee-b112c8b3b784\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6l557" Jul 18 02:40:33.876515 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.876051 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52dc3a30-b4eb-40ce-8380-9fc3a18db5df-tmp\") pod \"insights-operator-555d9d7885-d5t6p\" (UID: \"52dc3a30-b4eb-40ce-8380-9fc3a18db5df\") " pod="openshift-insights/insights-operator-555d9d7885-d5t6p" Jul 18 02:40:33.876515 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.876094 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/079022d6-f624-49c4-93a6-7bdb4b6a9ffd-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-scxbj\" (UID: \"079022d6-f624-49c4-93a6-7bdb4b6a9ffd\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-scxbj" Jul 18 02:40:33.876515 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.876413 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/709575dc-6eae-43e1-9208-cc228795929a-trusted-ca\") pod \"console-operator-565b7b569d-krtsg\" (UID: \"709575dc-6eae-43e1-9208-cc228795929a\") " pod="openshift-console-operator/console-operator-565b7b569d-krtsg" Jul 18 02:40:33.876698 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.876652 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/079022d6-f624-49c4-93a6-7bdb4b6a9ffd-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-scxbj\" (UID: \"079022d6-f624-49c4-93a6-7bdb4b6a9ffd\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-scxbj" Jul 18 02:40:33.878447 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.878426 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/53d15949-a9fb-4897-bec5-feb443eb99d9-serving-cert\") pod \"service-ca-operator-5bcbc94c78-gchp4\" (UID: \"53d15949-a9fb-4897-bec5-feb443eb99d9\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-gchp4" Jul 18 02:40:33.878527 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.878513 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/709575dc-6eae-43e1-9208-cc228795929a-serving-cert\") pod \"console-operator-565b7b569d-krtsg\" (UID: \"709575dc-6eae-43e1-9208-cc228795929a\") " pod="openshift-console-operator/console-operator-565b7b569d-krtsg" Jul 18 02:40:33.883095 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.883074 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7g52\" (UniqueName: \"kubernetes.io/projected/53d15949-a9fb-4897-bec5-feb443eb99d9-kube-api-access-h7g52\") pod \"service-ca-operator-5bcbc94c78-gchp4\" (UID: \"53d15949-a9fb-4897-bec5-feb443eb99d9\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-gchp4" Jul 18 02:40:33.886108 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.886084 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcfsw\" (UniqueName: \"kubernetes.io/projected/6fadc8c8-588f-41be-94e8-4c7de1ce8649-kube-api-access-bcfsw\") pod \"cluster-samples-operator-79b7c55f87-hjrpw\" (UID: \"6fadc8c8-588f-41be-94e8-4c7de1ce8649\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hjrpw" Jul 18 02:40:33.886763 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.886744 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzzpx\" (UniqueName: \"kubernetes.io/projected/709575dc-6eae-43e1-9208-cc228795929a-kube-api-access-bzzpx\") pod \"console-operator-565b7b569d-krtsg\" (UID: \"709575dc-6eae-43e1-9208-cc228795929a\") " pod="openshift-console-operator/console-operator-565b7b569d-krtsg" Jul 18 02:40:33.937110 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.937089 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-gchp4" Jul 18 02:40:33.968067 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.968041 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-krtsg" Jul 18 02:40:33.976566 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.976513 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2r9p\" (UniqueName: \"kubernetes.io/projected/52dc3a30-b4eb-40ce-8380-9fc3a18db5df-kube-api-access-d2r9p\") pod \"insights-operator-555d9d7885-d5t6p\" (UID: \"52dc3a30-b4eb-40ce-8380-9fc3a18db5df\") " pod="openshift-insights/insights-operator-555d9d7885-d5t6p" Jul 18 02:40:33.976566 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.976553 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f3a8a13-5e99-4e24-98ee-b112c8b3b784-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-6l557\" (UID: \"2f3a8a13-5e99-4e24-98ee-b112c8b3b784\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6l557" Jul 18 02:40:33.976721 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.976597 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52dc3a30-b4eb-40ce-8380-9fc3a18db5df-tmp\") pod \"insights-operator-555d9d7885-d5t6p\" (UID: \"52dc3a30-b4eb-40ce-8380-9fc3a18db5df\") " pod="openshift-insights/insights-operator-555d9d7885-d5t6p" Jul 18 02:40:33.976721 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.976642 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52dc3a30-b4eb-40ce-8380-9fc3a18db5df-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-d5t6p\" (UID: \"52dc3a30-b4eb-40ce-8380-9fc3a18db5df\") " pod="openshift-insights/insights-operator-555d9d7885-d5t6p" Jul 18 02:40:33.976721 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:33.976672 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 18 02:40:33.976721 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.976677 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp8vb\" (UniqueName: \"kubernetes.io/projected/9d34d35f-234b-4902-8a99-bc58ca2185de-kube-api-access-wp8vb\") pod \"volume-data-source-validator-78cf4c6f6-44wcx\" (UID: \"9d34d35f-234b-4902-8a99-bc58ca2185de\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-44wcx" Jul 18 02:40:33.976721 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.976711 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-metrics-certs\") pod \"router-default-789b768784-4p68t\" (UID: \"7ce9775c-ae6f-4891-b9df-58d8a04ec1c8\") " pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:40:33.976966 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:33.976729 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f3a8a13-5e99-4e24-98ee-b112c8b3b784-cluster-monitoring-operator-tls podName:2f3a8a13-5e99-4e24-98ee-b112c8b3b784 nodeName:}" failed. No retries permitted until 2026-07-18 02:40:34.476708958 +0000 UTC m=+58.080356635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2f3a8a13-5e99-4e24-98ee-b112c8b3b784-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-6l557" (UID: "2f3a8a13-5e99-4e24-98ee-b112c8b3b784") : secret "cluster-monitoring-operator-tls" not found Jul 18 02:40:33.976966 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.976780 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp4z6\" (UniqueName: \"kubernetes.io/projected/8353a5f3-694a-40ac-bde3-a22bc16e47b0-kube-api-access-xp4z6\") pod \"network-check-source-7bc5b6bf85-27nmz\" (UID: \"8353a5f3-694a-40ac-bde3-a22bc16e47b0\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-27nmz" Jul 18 02:40:33.976966 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:33.976790 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 18 02:40:33.976966 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.976816 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-service-ca-bundle\") pod \"router-default-789b768784-4p68t\" (UID: \"7ce9775c-ae6f-4891-b9df-58d8a04ec1c8\") " pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:40:33.976966 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:33.976838 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-metrics-certs podName:7ce9775c-ae6f-4891-b9df-58d8a04ec1c8 nodeName:}" failed. No retries permitted until 2026-07-18 02:40:34.476822099 +0000 UTC m=+58.080469779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-metrics-certs") pod "router-default-789b768784-4p68t" (UID: "7ce9775c-ae6f-4891-b9df-58d8a04ec1c8") : secret "router-metrics-certs-default" not found Jul 18 02:40:33.976966 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.976868 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-default-certificate\") pod \"router-default-789b768784-4p68t\" (UID: \"7ce9775c-ae6f-4891-b9df-58d8a04ec1c8\") " pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:40:33.976966 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:33.976892 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-service-ca-bundle podName:7ce9775c-ae6f-4891-b9df-58d8a04ec1c8 nodeName:}" failed. No retries permitted until 2026-07-18 02:40:34.476880508 +0000 UTC m=+58.080528178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-service-ca-bundle") pod "router-default-789b768784-4p68t" (UID: "7ce9775c-ae6f-4891-b9df-58d8a04ec1c8") : configmap references non-existent config key: service-ca.crt Jul 18 02:40:33.976966 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.976935 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzzdl\" (UniqueName: \"kubernetes.io/projected/2f3a8a13-5e99-4e24-98ee-b112c8b3b784-kube-api-access-jzzdl\") pod \"cluster-monitoring-operator-76cf8966f8-6l557\" (UID: \"2f3a8a13-5e99-4e24-98ee-b112c8b3b784\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6l557" Jul 18 02:40:33.976966 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.976960 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/52dc3a30-b4eb-40ce-8380-9fc3a18db5df-snapshots\") pod \"insights-operator-555d9d7885-d5t6p\" (UID: \"52dc3a30-b4eb-40ce-8380-9fc3a18db5df\") " pod="openshift-insights/insights-operator-555d9d7885-d5t6p" Jul 18 02:40:33.977434 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.976982 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drh7b\" (UniqueName: \"kubernetes.io/projected/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-kube-api-access-drh7b\") pod \"router-default-789b768784-4p68t\" (UID: \"7ce9775c-ae6f-4891-b9df-58d8a04ec1c8\") " pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:40:33.977434 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.977009 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-stats-auth\") pod \"router-default-789b768784-4p68t\" (UID: \"7ce9775c-ae6f-4891-b9df-58d8a04ec1c8\") " pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:40:33.977434 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.977066 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52dc3a30-b4eb-40ce-8380-9fc3a18db5df-tmp\") pod \"insights-operator-555d9d7885-d5t6p\" (UID: \"52dc3a30-b4eb-40ce-8380-9fc3a18db5df\") " pod="openshift-insights/insights-operator-555d9d7885-d5t6p" Jul 18 02:40:33.977584 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.977473 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e05161ba-27ca-4679-9439-bad0efa4500f-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-c4l7k\" (UID: \"e05161ba-27ca-4679-9439-bad0efa4500f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-c4l7k" Jul 18 02:40:33.977584 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.977507 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52dc3a30-b4eb-40ce-8380-9fc3a18db5df-service-ca-bundle\") pod \"insights-operator-555d9d7885-d5t6p\" (UID: \"52dc3a30-b4eb-40ce-8380-9fc3a18db5df\") " pod="openshift-insights/insights-operator-555d9d7885-d5t6p" Jul 18 02:40:33.977584 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.977515 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/52dc3a30-b4eb-40ce-8380-9fc3a18db5df-snapshots\") pod \"insights-operator-555d9d7885-d5t6p\" (UID: \"52dc3a30-b4eb-40ce-8380-9fc3a18db5df\") " pod="openshift-insights/insights-operator-555d9d7885-d5t6p" Jul 18 02:40:33.977584 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.977561 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2f3a8a13-5e99-4e24-98ee-b112c8b3b784-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-6l557\" (UID: \"2f3a8a13-5e99-4e24-98ee-b112c8b3b784\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6l557" Jul 18 02:40:33.977776 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.977603 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e05161ba-27ca-4679-9439-bad0efa4500f-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-c4l7k\" (UID: \"e05161ba-27ca-4679-9439-bad0efa4500f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-c4l7k" Jul 18 02:40:33.977776 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.977629 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qt2t\" (UniqueName: \"kubernetes.io/projected/e05161ba-27ca-4679-9439-bad0efa4500f-kube-api-access-9qt2t\") pod \"kube-storage-version-migrator-operator-7658d9b749-c4l7k\" (UID: \"e05161ba-27ca-4679-9439-bad0efa4500f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-c4l7k" Jul 18 02:40:33.977776 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.977673 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/52dc3a30-b4eb-40ce-8380-9fc3a18db5df-serving-cert\") pod \"insights-operator-555d9d7885-d5t6p\" (UID: \"52dc3a30-b4eb-40ce-8380-9fc3a18db5df\") " pod="openshift-insights/insights-operator-555d9d7885-d5t6p" Jul 18 02:40:33.977776 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.977710 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52dc3a30-b4eb-40ce-8380-9fc3a18db5df-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-d5t6p\" (UID: \"52dc3a30-b4eb-40ce-8380-9fc3a18db5df\") " pod="openshift-insights/insights-operator-555d9d7885-d5t6p" Jul 18 02:40:33.978373 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.978083 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e05161ba-27ca-4679-9439-bad0efa4500f-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-c4l7k\" (UID: \"e05161ba-27ca-4679-9439-bad0efa4500f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-c4l7k" Jul 18 02:40:33.978494 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.978472 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2f3a8a13-5e99-4e24-98ee-b112c8b3b784-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-6l557\" (UID: \"2f3a8a13-5e99-4e24-98ee-b112c8b3b784\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6l557" Jul 18 02:40:33.978895 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.978871 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52dc3a30-b4eb-40ce-8380-9fc3a18db5df-service-ca-bundle\") pod \"insights-operator-555d9d7885-d5t6p\" (UID: \"52dc3a30-b4eb-40ce-8380-9fc3a18db5df\") " pod="openshift-insights/insights-operator-555d9d7885-d5t6p" Jul 18 02:40:33.980564 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.980222 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-stats-auth\") pod \"router-default-789b768784-4p68t\" (UID: \"7ce9775c-ae6f-4891-b9df-58d8a04ec1c8\") " pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:40:33.980564 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.980474 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/52dc3a30-b4eb-40ce-8380-9fc3a18db5df-serving-cert\") pod \"insights-operator-555d9d7885-d5t6p\" (UID: \"52dc3a30-b4eb-40ce-8380-9fc3a18db5df\") " pod="openshift-insights/insights-operator-555d9d7885-d5t6p" Jul 18 02:40:33.980648 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.980627 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-default-certificate\") pod \"router-default-789b768784-4p68t\" (UID: \"7ce9775c-ae6f-4891-b9df-58d8a04ec1c8\") " pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:40:33.980975 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.980955 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e05161ba-27ca-4679-9439-bad0efa4500f-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-c4l7k\" (UID: \"e05161ba-27ca-4679-9439-bad0efa4500f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-c4l7k" Jul 18 02:40:33.990423 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.990398 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp4z6\" (UniqueName: \"kubernetes.io/projected/8353a5f3-694a-40ac-bde3-a22bc16e47b0-kube-api-access-xp4z6\") pod \"network-check-source-7bc5b6bf85-27nmz\" (UID: \"8353a5f3-694a-40ac-bde3-a22bc16e47b0\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-27nmz" Jul 18 02:40:33.991247 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.991175 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qt2t\" (UniqueName: \"kubernetes.io/projected/e05161ba-27ca-4679-9439-bad0efa4500f-kube-api-access-9qt2t\") pod \"kube-storage-version-migrator-operator-7658d9b749-c4l7k\" (UID: \"e05161ba-27ca-4679-9439-bad0efa4500f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-c4l7k" Jul 18 02:40:33.991921 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.991880 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzzdl\" (UniqueName: \"kubernetes.io/projected/2f3a8a13-5e99-4e24-98ee-b112c8b3b784-kube-api-access-jzzdl\") pod \"cluster-monitoring-operator-76cf8966f8-6l557\" (UID: \"2f3a8a13-5e99-4e24-98ee-b112c8b3b784\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6l557" Jul 18 02:40:33.992210 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.992167 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drh7b\" (UniqueName: \"kubernetes.io/projected/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-kube-api-access-drh7b\") pod \"router-default-789b768784-4p68t\" (UID: \"7ce9775c-ae6f-4891-b9df-58d8a04ec1c8\") " pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:40:33.994032 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:33.994013 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2r9p\" (UniqueName: \"kubernetes.io/projected/52dc3a30-b4eb-40ce-8380-9fc3a18db5df-kube-api-access-d2r9p\") pod \"insights-operator-555d9d7885-d5t6p\" (UID: \"52dc3a30-b4eb-40ce-8380-9fc3a18db5df\") " pod="openshift-insights/insights-operator-555d9d7885-d5t6p" Jul 18 02:40:34.038851 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:34.038818 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-27nmz" Jul 18 02:40:34.050470 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:34.050442 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-c4l7k" Jul 18 02:40:34.064604 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:34.064389 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-d5t6p" Jul 18 02:40:34.078121 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:34.077648 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-gchp4"] Jul 18 02:40:34.078304 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:34.078282 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp8vb\" (UniqueName: \"kubernetes.io/projected/9d34d35f-234b-4902-8a99-bc58ca2185de-kube-api-access-wp8vb\") pod \"volume-data-source-validator-78cf4c6f6-44wcx\" (UID: \"9d34d35f-234b-4902-8a99-bc58ca2185de\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-44wcx" Jul 18 02:40:34.080427 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:40:34.080374 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53d15949_a9fb_4897_bec5_feb443eb99d9.slice/crio-beea59d1480cbcbaade375fe1a43bfbf1bb434fcad42f60aa56a49b9ee75fe00 WatchSource:0}: Error finding container beea59d1480cbcbaade375fe1a43bfbf1bb434fcad42f60aa56a49b9ee75fe00: Status 404 returned error can't find the container with id beea59d1480cbcbaade375fe1a43bfbf1bb434fcad42f60aa56a49b9ee75fe00 Jul 18 02:40:34.087876 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:34.087825 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp8vb\" (UniqueName: \"kubernetes.io/projected/9d34d35f-234b-4902-8a99-bc58ca2185de-kube-api-access-wp8vb\") pod \"volume-data-source-validator-78cf4c6f6-44wcx\" (UID: \"9d34d35f-234b-4902-8a99-bc58ca2185de\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-44wcx" Jul 18 02:40:34.096579 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:34.096523 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-krtsg"] Jul 18 02:40:34.139144 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:34.139082 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-44wcx" Jul 18 02:40:34.186923 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:34.186879 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-27nmz"] Jul 18 02:40:34.191420 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:40:34.191370 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8353a5f3_694a_40ac_bde3_a22bc16e47b0.slice/crio-d3e7b88b471f512dd40bc78b7011e45a5f6ef65a14f81a4450e4057cb2dd5765 WatchSource:0}: Error finding container d3e7b88b471f512dd40bc78b7011e45a5f6ef65a14f81a4450e4057cb2dd5765: Status 404 returned error can't find the container with id d3e7b88b471f512dd40bc78b7011e45a5f6ef65a14f81a4450e4057cb2dd5765 Jul 18 02:40:34.246635 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:34.246598 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-gchp4" event={"ID":"53d15949-a9fb-4897-bec5-feb443eb99d9","Type":"ContainerStarted","Data":"beea59d1480cbcbaade375fe1a43bfbf1bb434fcad42f60aa56a49b9ee75fe00"} Jul 18 02:40:34.247678 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:34.247652 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-krtsg" event={"ID":"709575dc-6eae-43e1-9208-cc228795929a","Type":"ContainerStarted","Data":"c3770d6bc22b8898e7e6c42ebb162a95d91dda27a44c024d5191f91209f56297"} Jul 18 02:40:34.248666 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:34.248642 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-27nmz" event={"ID":"8353a5f3-694a-40ac-bde3-a22bc16e47b0","Type":"ContainerStarted","Data":"d3e7b88b471f512dd40bc78b7011e45a5f6ef65a14f81a4450e4057cb2dd5765"} Jul 18 02:40:34.266527 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:34.266445 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-44wcx"] Jul 18 02:40:34.268315 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:40:34.268291 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d34d35f_234b_4902_8a99_bc58ca2185de.slice/crio-e63c1047ecfff06377405c9f47a3ef94608a19095c9bc17e235872815676d178 WatchSource:0}: Error finding container e63c1047ecfff06377405c9f47a3ef94608a19095c9bc17e235872815676d178: Status 404 returned error can't find the container with id e63c1047ecfff06377405c9f47a3ef94608a19095c9bc17e235872815676d178 Jul 18 02:40:34.381852 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:34.381802 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fadc8c8-588f-41be-94e8-4c7de1ce8649-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-hjrpw\" (UID: \"6fadc8c8-588f-41be-94e8-4c7de1ce8649\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hjrpw" Jul 18 02:40:34.381852 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:34.381837 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/079022d6-f624-49c4-93a6-7bdb4b6a9ffd-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-scxbj\" (UID: \"079022d6-f624-49c4-93a6-7bdb4b6a9ffd\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-scxbj" Jul 18 02:40:34.382012 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:34.381963 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 18 02:40:34.385504 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:34.382392 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fadc8c8-588f-41be-94e8-4c7de1ce8649-samples-operator-tls podName:6fadc8c8-588f-41be-94e8-4c7de1ce8649 nodeName:}" failed. No retries permitted until 2026-07-18 02:40:35.382365573 +0000 UTC m=+58.986013251 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6fadc8c8-588f-41be-94e8-4c7de1ce8649-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-hjrpw" (UID: "6fadc8c8-588f-41be-94e8-4c7de1ce8649") : secret "samples-operator-tls" not found Jul 18 02:40:34.385504 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:34.382399 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 18 02:40:34.385504 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:34.382461 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/079022d6-f624-49c4-93a6-7bdb4b6a9ffd-networking-console-plugin-cert podName:079022d6-f624-49c4-93a6-7bdb4b6a9ffd nodeName:}" failed. No retries permitted until 2026-07-18 02:40:35.382449598 +0000 UTC m=+58.986097262 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/079022d6-f624-49c4-93a6-7bdb4b6a9ffd-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-scxbj" (UID: "079022d6-f624-49c4-93a6-7bdb4b6a9ffd") : secret "networking-console-plugin-cert" not found Jul 18 02:40:34.412254 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:34.412230 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-c4l7k"] Jul 18 02:40:34.415227 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:34.415169 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-d5t6p"] Jul 18 02:40:34.415416 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:40:34.415392 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode05161ba_27ca_4679_9439_bad0efa4500f.slice/crio-8dab2875597a19e6318b3d0f3a8965cc7c0b8daedf84194c87ac8e81b7245170 WatchSource:0}: Error finding container 8dab2875597a19e6318b3d0f3a8965cc7c0b8daedf84194c87ac8e81b7245170: Status 404 returned error can't find the container with id 8dab2875597a19e6318b3d0f3a8965cc7c0b8daedf84194c87ac8e81b7245170 Jul 18 02:40:34.417910 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:40:34.417890 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52dc3a30_b4eb_40ce_8380_9fc3a18db5df.slice/crio-db05f7a53991f85d19da9d59072c2832d2fb1cb2f0734e64f9c6f185d1df2357 WatchSource:0}: Error finding container db05f7a53991f85d19da9d59072c2832d2fb1cb2f0734e64f9c6f185d1df2357: Status 404 returned error can't find the container with id db05f7a53991f85d19da9d59072c2832d2fb1cb2f0734e64f9c6f185d1df2357 Jul 18 02:40:34.483333 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:34.483311 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f3a8a13-5e99-4e24-98ee-b112c8b3b784-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-6l557\" (UID: \"2f3a8a13-5e99-4e24-98ee-b112c8b3b784\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6l557" Jul 18 02:40:34.483446 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:34.483369 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-metrics-certs\") pod \"router-default-789b768784-4p68t\" (UID: \"7ce9775c-ae6f-4891-b9df-58d8a04ec1c8\") " pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:40:34.483446 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:34.483410 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-service-ca-bundle\") pod \"router-default-789b768784-4p68t\" (UID: \"7ce9775c-ae6f-4891-b9df-58d8a04ec1c8\") " pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:40:34.483550 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:34.483474 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 18 02:40:34.483550 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:34.483517 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-service-ca-bundle podName:7ce9775c-ae6f-4891-b9df-58d8a04ec1c8 nodeName:}" failed. No retries permitted until 2026-07-18 02:40:35.483500607 +0000 UTC m=+59.087148291 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-service-ca-bundle") pod "router-default-789b768784-4p68t" (UID: "7ce9775c-ae6f-4891-b9df-58d8a04ec1c8") : configmap references non-existent config key: service-ca.crt Jul 18 02:40:34.483550 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:34.483530 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f3a8a13-5e99-4e24-98ee-b112c8b3b784-cluster-monitoring-operator-tls podName:2f3a8a13-5e99-4e24-98ee-b112c8b3b784 nodeName:}" failed. No retries permitted until 2026-07-18 02:40:35.483525035 +0000 UTC m=+59.087172700 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2f3a8a13-5e99-4e24-98ee-b112c8b3b784-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-6l557" (UID: "2f3a8a13-5e99-4e24-98ee-b112c8b3b784") : secret "cluster-monitoring-operator-tls" not found Jul 18 02:40:34.483550 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:34.483536 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 18 02:40:34.483733 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:34.483586 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-metrics-certs podName:7ce9775c-ae6f-4891-b9df-58d8a04ec1c8 nodeName:}" failed. No retries permitted until 2026-07-18 02:40:35.483573787 +0000 UTC m=+59.087221452 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-metrics-certs") pod "router-default-789b768784-4p68t" (UID: "7ce9775c-ae6f-4891-b9df-58d8a04ec1c8") : secret "router-metrics-certs-default" not found Jul 18 02:40:35.254734 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:35.254643 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-44wcx" event={"ID":"9d34d35f-234b-4902-8a99-bc58ca2185de","Type":"ContainerStarted","Data":"e63c1047ecfff06377405c9f47a3ef94608a19095c9bc17e235872815676d178"} Jul 18 02:40:35.258347 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:35.258272 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-d5t6p" event={"ID":"52dc3a30-b4eb-40ce-8380-9fc3a18db5df","Type":"ContainerStarted","Data":"db05f7a53991f85d19da9d59072c2832d2fb1cb2f0734e64f9c6f185d1df2357"} Jul 18 02:40:35.260174 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:35.260115 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-c4l7k" event={"ID":"e05161ba-27ca-4679-9439-bad0efa4500f","Type":"ContainerStarted","Data":"8dab2875597a19e6318b3d0f3a8965cc7c0b8daedf84194c87ac8e81b7245170"} Jul 18 02:40:35.391214 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:35.390393 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fadc8c8-588f-41be-94e8-4c7de1ce8649-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-hjrpw\" (UID: \"6fadc8c8-588f-41be-94e8-4c7de1ce8649\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hjrpw" Jul 18 02:40:35.391214 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:35.390455 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/079022d6-f624-49c4-93a6-7bdb4b6a9ffd-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-scxbj\" (UID: \"079022d6-f624-49c4-93a6-7bdb4b6a9ffd\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-scxbj" Jul 18 02:40:35.391214 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:35.390666 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 18 02:40:35.391214 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:35.390724 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/079022d6-f624-49c4-93a6-7bdb4b6a9ffd-networking-console-plugin-cert podName:079022d6-f624-49c4-93a6-7bdb4b6a9ffd nodeName:}" failed. No retries permitted until 2026-07-18 02:40:37.390707747 +0000 UTC m=+60.994355424 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/079022d6-f624-49c4-93a6-7bdb4b6a9ffd-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-scxbj" (UID: "079022d6-f624-49c4-93a6-7bdb4b6a9ffd") : secret "networking-console-plugin-cert" not found Jul 18 02:40:35.391214 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:35.391120 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 18 02:40:35.391214 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:35.391167 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fadc8c8-588f-41be-94e8-4c7de1ce8649-samples-operator-tls podName:6fadc8c8-588f-41be-94e8-4c7de1ce8649 nodeName:}" failed. No retries permitted until 2026-07-18 02:40:37.391152599 +0000 UTC m=+60.994800269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6fadc8c8-588f-41be-94e8-4c7de1ce8649-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-hjrpw" (UID: "6fadc8c8-588f-41be-94e8-4c7de1ce8649") : secret "samples-operator-tls" not found Jul 18 02:40:35.491950 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:35.491341 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f3a8a13-5e99-4e24-98ee-b112c8b3b784-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-6l557\" (UID: \"2f3a8a13-5e99-4e24-98ee-b112c8b3b784\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6l557" Jul 18 02:40:35.491950 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:35.491444 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-metrics-certs\") pod \"router-default-789b768784-4p68t\" (UID: \"7ce9775c-ae6f-4891-b9df-58d8a04ec1c8\") " pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:40:35.491950 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:35.491492 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-service-ca-bundle\") pod \"router-default-789b768784-4p68t\" (UID: \"7ce9775c-ae6f-4891-b9df-58d8a04ec1c8\") " pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:40:35.491950 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:35.491840 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 18 02:40:35.491950 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:35.491878 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 18 02:40:35.492577 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:35.492473 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-service-ca-bundle podName:7ce9775c-ae6f-4891-b9df-58d8a04ec1c8 nodeName:}" failed. No retries permitted until 2026-07-18 02:40:37.491704285 +0000 UTC m=+61.095351965 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-service-ca-bundle") pod "router-default-789b768784-4p68t" (UID: "7ce9775c-ae6f-4891-b9df-58d8a04ec1c8") : configmap references non-existent config key: service-ca.crt Jul 18 02:40:35.492577 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:35.492516 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f3a8a13-5e99-4e24-98ee-b112c8b3b784-cluster-monitoring-operator-tls podName:2f3a8a13-5e99-4e24-98ee-b112c8b3b784 nodeName:}" failed. No retries permitted until 2026-07-18 02:40:37.492496669 +0000 UTC m=+61.096144341 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2f3a8a13-5e99-4e24-98ee-b112c8b3b784-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-6l557" (UID: "2f3a8a13-5e99-4e24-98ee-b112c8b3b784") : secret "cluster-monitoring-operator-tls" not found Jul 18 02:40:35.492577 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:35.492531 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-metrics-certs podName:7ce9775c-ae6f-4891-b9df-58d8a04ec1c8 nodeName:}" failed. No retries permitted until 2026-07-18 02:40:37.492522633 +0000 UTC m=+61.096170299 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-metrics-certs") pod "router-default-789b768784-4p68t" (UID: "7ce9775c-ae6f-4891-b9df-58d8a04ec1c8") : secret "router-metrics-certs-default" not found Jul 18 02:40:37.164405 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:37.164373 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kpl5z" Jul 18 02:40:37.408021 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:37.407989 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fadc8c8-588f-41be-94e8-4c7de1ce8649-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-hjrpw\" (UID: \"6fadc8c8-588f-41be-94e8-4c7de1ce8649\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hjrpw" Jul 18 02:40:37.408156 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:37.408034 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/079022d6-f624-49c4-93a6-7bdb4b6a9ffd-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-scxbj\" (UID: \"079022d6-f624-49c4-93a6-7bdb4b6a9ffd\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-scxbj" Jul 18 02:40:37.408156 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:37.408137 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 18 02:40:37.408237 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:37.408176 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 18 02:40:37.408237 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:37.408219 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fadc8c8-588f-41be-94e8-4c7de1ce8649-samples-operator-tls podName:6fadc8c8-588f-41be-94e8-4c7de1ce8649 nodeName:}" failed. No retries permitted until 2026-07-18 02:40:41.408199789 +0000 UTC m=+65.011847467 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6fadc8c8-588f-41be-94e8-4c7de1ce8649-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-hjrpw" (UID: "6fadc8c8-588f-41be-94e8-4c7de1ce8649") : secret "samples-operator-tls" not found Jul 18 02:40:37.408306 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:37.408241 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/079022d6-f624-49c4-93a6-7bdb4b6a9ffd-networking-console-plugin-cert podName:079022d6-f624-49c4-93a6-7bdb4b6a9ffd nodeName:}" failed. No retries permitted until 2026-07-18 02:40:41.408229354 +0000 UTC m=+65.011877018 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/079022d6-f624-49c4-93a6-7bdb4b6a9ffd-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-scxbj" (UID: "079022d6-f624-49c4-93a6-7bdb4b6a9ffd") : secret "networking-console-plugin-cert" not found Jul 18 02:40:37.508790 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:37.508763 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f3a8a13-5e99-4e24-98ee-b112c8b3b784-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-6l557\" (UID: \"2f3a8a13-5e99-4e24-98ee-b112c8b3b784\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6l557" Jul 18 02:40:37.508893 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:37.508819 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-metrics-certs\") pod \"router-default-789b768784-4p68t\" (UID: \"7ce9775c-ae6f-4891-b9df-58d8a04ec1c8\") " pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:40:37.508945 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:37.508891 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 18 02:40:37.508945 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:37.508922 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-service-ca-bundle\") pod \"router-default-789b768784-4p68t\" (UID: \"7ce9775c-ae6f-4891-b9df-58d8a04ec1c8\") " pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:40:37.508945 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:37.508936 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-metrics-certs podName:7ce9775c-ae6f-4891-b9df-58d8a04ec1c8 nodeName:}" failed. No retries permitted until 2026-07-18 02:40:41.508923273 +0000 UTC m=+65.112570938 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-metrics-certs") pod "router-default-789b768784-4p68t" (UID: "7ce9775c-ae6f-4891-b9df-58d8a04ec1c8") : secret "router-metrics-certs-default" not found Jul 18 02:40:37.509041 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:37.508892 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 18 02:40:37.509041 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:37.509015 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-service-ca-bundle podName:7ce9775c-ae6f-4891-b9df-58d8a04ec1c8 nodeName:}" failed. No retries permitted until 2026-07-18 02:40:41.509004442 +0000 UTC m=+65.112652123 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-service-ca-bundle") pod "router-default-789b768784-4p68t" (UID: "7ce9775c-ae6f-4891-b9df-58d8a04ec1c8") : configmap references non-existent config key: service-ca.crt Jul 18 02:40:37.509041 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:37.509029 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f3a8a13-5e99-4e24-98ee-b112c8b3b784-cluster-monitoring-operator-tls podName:2f3a8a13-5e99-4e24-98ee-b112c8b3b784 nodeName:}" failed. No retries permitted until 2026-07-18 02:40:41.509023804 +0000 UTC m=+65.112671468 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2f3a8a13-5e99-4e24-98ee-b112c8b3b784-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-6l557" (UID: "2f3a8a13-5e99-4e24-98ee-b112c8b3b784") : secret "cluster-monitoring-operator-tls" not found Jul 18 02:40:40.282347 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.282311 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-krtsg_709575dc-6eae-43e1-9208-cc228795929a/console-operator/0.log" Jul 18 02:40:40.282800 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.282357 2585 generic.go:356] "Generic (PLEG): container finished" podID="709575dc-6eae-43e1-9208-cc228795929a" containerID="0e75a2e8c6e6dcaf23021308f22f1f2edf2a84aa69239501a3ed641651151efc" exitCode=255 Jul 18 02:40:40.282800 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.282598 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-krtsg" event={"ID":"709575dc-6eae-43e1-9208-cc228795929a","Type":"ContainerDied","Data":"0e75a2e8c6e6dcaf23021308f22f1f2edf2a84aa69239501a3ed641651151efc"} Jul 18 02:40:40.282800 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.282691 2585 scope.go:117] "RemoveContainer" containerID="0e75a2e8c6e6dcaf23021308f22f1f2edf2a84aa69239501a3ed641651151efc" Jul 18 02:40:40.284657 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.284630 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-d5t6p" event={"ID":"52dc3a30-b4eb-40ce-8380-9fc3a18db5df","Type":"ContainerStarted","Data":"295c34e61a24d6023ce5028ca9778112dea6dce45ed13d2748443c4e13150f90"} Jul 18 02:40:40.286885 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.286862 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-c4l7k" event={"ID":"e05161ba-27ca-4679-9439-bad0efa4500f","Type":"ContainerStarted","Data":"2f5e2ce5a278080654560df39b2942932e44412124bf9d24142362a24a1c84e2"} Jul 18 02:40:40.288707 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.288661 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-27nmz" event={"ID":"8353a5f3-694a-40ac-bde3-a22bc16e47b0","Type":"ContainerStarted","Data":"cdf2cccc21fa4f2760c8ed0f3c2755177910e37430723f47463d597cafe0e31f"} Jul 18 02:40:40.290333 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.290307 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-gchp4" event={"ID":"53d15949-a9fb-4897-bec5-feb443eb99d9","Type":"ContainerStarted","Data":"5f231e47886c80536b57cdd11dccee55a3c44756819165654a03d98778be9e51"} Jul 18 02:40:40.292590 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.292218 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-44wcx" event={"ID":"9d34d35f-234b-4902-8a99-bc58ca2185de","Type":"ContainerStarted","Data":"aa2968f6ca7dbc1a7176836035c7f3ea066aa5fb30dd933b90862a95e0f3ff33"} Jul 18 02:40:40.335900 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.335716 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-d5t6p" podStartSLOduration=2.251365271 podStartE2EDuration="7.335697373s" totalImagesPullingTime="5.084332102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:40:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:40:40.317951999 +0000 UTC m=+63.921599687" watchObservedRunningTime="2026-07-18 02:40:40.335697373 +0000 UTC m=+63.939345060" Jul 18 02:40:40.336875 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.336616 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-gchp4" podStartSLOduration=1.915759907 podStartE2EDuration="7.336604475s" totalImagesPullingTime="5.420844568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:40:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:40:40.334876639 +0000 UTC m=+63.938524323" watchObservedRunningTime="2026-07-18 02:40:40.336604475 +0000 UTC m=+63.940252157" Jul 18 02:40:40.351981 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.351915 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-27nmz" podStartSLOduration=1.9940397330000001 podStartE2EDuration="7.351897184s" totalImagesPullingTime="5.357857451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:40:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:40:40.351000592 +0000 UTC m=+63.954648274" watchObservedRunningTime="2026-07-18 02:40:40.351897184 +0000 UTC m=+63.955544873" Jul 18 02:40:40.369729 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.369689 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-c4l7k" podStartSLOduration=2.25343964 podStartE2EDuration="7.369674024s" totalImagesPullingTime="5.116234384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:40:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:40:40.368884715 +0000 UTC m=+63.972532402" watchObservedRunningTime="2026-07-18 02:40:40.369674024 +0000 UTC m=+63.973321711" Jul 18 02:40:40.389358 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.389314 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-44wcx" podStartSLOduration=2.161875075 podStartE2EDuration="7.389296167s" totalImagesPullingTime="5.227421092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:40:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:40:40.387784467 +0000 UTC m=+63.991432153" watchObservedRunningTime="2026-07-18 02:40:40.389296167 +0000 UTC m=+63.992943855" Jul 18 02:40:40.667750 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.667717 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fvt9x"] Jul 18 02:40:40.671442 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.671420 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fvt9x" Jul 18 02:40:40.674410 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.674386 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8nv5k\"" Jul 18 02:40:40.674538 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.674494 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 18 02:40:40.674818 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.674802 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 18 02:40:40.682981 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.682956 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fvt9x"] Jul 18 02:40:40.739991 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.739965 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-crio-socket\") pod \"insights-runtime-extractor-fvt9x\" (UID: \"f7b220b6-d648-4f31-b11b-2d3d7be9a13a\") " pod="openshift-insights/insights-runtime-extractor-fvt9x" Jul 18 02:40:40.740128 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.740097 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fvt9x\" (UID: \"f7b220b6-d648-4f31-b11b-2d3d7be9a13a\") " pod="openshift-insights/insights-runtime-extractor-fvt9x" Jul 18 02:40:40.740198 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.740135 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fvt9x\" (UID: \"f7b220b6-d648-4f31-b11b-2d3d7be9a13a\") " pod="openshift-insights/insights-runtime-extractor-fvt9x" Jul 18 02:40:40.740261 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.740200 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbg42\" (UniqueName: \"kubernetes.io/projected/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-kube-api-access-mbg42\") pod \"insights-runtime-extractor-fvt9x\" (UID: \"f7b220b6-d648-4f31-b11b-2d3d7be9a13a\") " pod="openshift-insights/insights-runtime-extractor-fvt9x" Jul 18 02:40:40.740331 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.740316 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-data-volume\") pod \"insights-runtime-extractor-fvt9x\" (UID: \"f7b220b6-d648-4f31-b11b-2d3d7be9a13a\") " pod="openshift-insights/insights-runtime-extractor-fvt9x" Jul 18 02:40:40.841635 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.841576 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbg42\" (UniqueName: \"kubernetes.io/projected/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-kube-api-access-mbg42\") pod \"insights-runtime-extractor-fvt9x\" (UID: \"f7b220b6-d648-4f31-b11b-2d3d7be9a13a\") " pod="openshift-insights/insights-runtime-extractor-fvt9x" Jul 18 02:40:40.841759 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.841698 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-data-volume\") pod \"insights-runtime-extractor-fvt9x\" (UID: \"f7b220b6-d648-4f31-b11b-2d3d7be9a13a\") " pod="openshift-insights/insights-runtime-extractor-fvt9x" Jul 18 02:40:40.841759 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.841736 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-crio-socket\") pod \"insights-runtime-extractor-fvt9x\" (UID: \"f7b220b6-d648-4f31-b11b-2d3d7be9a13a\") " pod="openshift-insights/insights-runtime-extractor-fvt9x" Jul 18 02:40:40.841858 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.841824 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fvt9x\" (UID: \"f7b220b6-d648-4f31-b11b-2d3d7be9a13a\") " pod="openshift-insights/insights-runtime-extractor-fvt9x" Jul 18 02:40:40.841903 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.841855 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fvt9x\" (UID: \"f7b220b6-d648-4f31-b11b-2d3d7be9a13a\") " pod="openshift-insights/insights-runtime-extractor-fvt9x" Jul 18 02:40:40.841939 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.841921 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-crio-socket\") pod \"insights-runtime-extractor-fvt9x\" (UID: \"f7b220b6-d648-4f31-b11b-2d3d7be9a13a\") " pod="openshift-insights/insights-runtime-extractor-fvt9x" Jul 18 02:40:40.841970 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:40.841933 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 18 02:40:40.841970 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:40.841960 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 02:40:40.842037 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:40.842017 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-insights-runtime-extractor-tls podName:f7b220b6-d648-4f31-b11b-2d3d7be9a13a nodeName:}" failed. No retries permitted until 2026-07-18 02:40:41.341999308 +0000 UTC m=+64.945646979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fvt9x" (UID: "f7b220b6-d648-4f31-b11b-2d3d7be9a13a") : secret "insights-runtime-extractor-tls" not found Jul 18 02:40:40.842083 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:40.842036 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-kube-rbac-proxy-cm podName:f7b220b6-d648-4f31-b11b-2d3d7be9a13a nodeName:}" failed. No retries permitted until 2026-07-18 02:40:41.342026135 +0000 UTC m=+64.945673801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fvt9x" (UID: "f7b220b6-d648-4f31-b11b-2d3d7be9a13a") : configmap "kube-rbac-proxy" not found Jul 18 02:40:40.842083 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.842062 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-data-volume\") pod \"insights-runtime-extractor-fvt9x\" (UID: \"f7b220b6-d648-4f31-b11b-2d3d7be9a13a\") " pod="openshift-insights/insights-runtime-extractor-fvt9x" Jul 18 02:40:40.851499 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:40.851472 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbg42\" (UniqueName: \"kubernetes.io/projected/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-kube-api-access-mbg42\") pod \"insights-runtime-extractor-fvt9x\" (UID: \"f7b220b6-d648-4f31-b11b-2d3d7be9a13a\") " pod="openshift-insights/insights-runtime-extractor-fvt9x" Jul 18 02:40:41.297474 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:41.297447 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-krtsg_709575dc-6eae-43e1-9208-cc228795929a/console-operator/1.log" Jul 18 02:40:41.297951 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:41.297937 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-krtsg_709575dc-6eae-43e1-9208-cc228795929a/console-operator/0.log" Jul 18 02:40:41.298026 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:41.297971 2585 generic.go:356] "Generic (PLEG): container finished" podID="709575dc-6eae-43e1-9208-cc228795929a" containerID="9241824ff40ffcc74352a3d80d205262bdcce9d9593f0e014db64008a5bb29d1" exitCode=255 Jul 18 02:40:41.298101 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:41.298076 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-krtsg" event={"ID":"709575dc-6eae-43e1-9208-cc228795929a","Type":"ContainerDied","Data":"9241824ff40ffcc74352a3d80d205262bdcce9d9593f0e014db64008a5bb29d1"} Jul 18 02:40:41.298165 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:41.298148 2585 scope.go:117] "RemoveContainer" containerID="0e75a2e8c6e6dcaf23021308f22f1f2edf2a84aa69239501a3ed641651151efc" Jul 18 02:40:41.298331 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:41.298310 2585 scope.go:117] "RemoveContainer" containerID="9241824ff40ffcc74352a3d80d205262bdcce9d9593f0e014db64008a5bb29d1" Jul 18 02:40:41.298535 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:41.298510 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-krtsg_openshift-console-operator(709575dc-6eae-43e1-9208-cc228795929a)\"" pod="openshift-console-operator/console-operator-565b7b569d-krtsg" podUID="709575dc-6eae-43e1-9208-cc228795929a" Jul 18 02:40:41.346108 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:41.346082 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fvt9x\" (UID: \"f7b220b6-d648-4f31-b11b-2d3d7be9a13a\") " pod="openshift-insights/insights-runtime-extractor-fvt9x" Jul 18 02:40:41.346211 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:41.346115 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fvt9x\" (UID: \"f7b220b6-d648-4f31-b11b-2d3d7be9a13a\") " pod="openshift-insights/insights-runtime-extractor-fvt9x" Jul 18 02:40:41.346264 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:41.346208 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 18 02:40:41.346264 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:41.346234 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 02:40:41.346264 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:41.346260 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-kube-rbac-proxy-cm podName:f7b220b6-d648-4f31-b11b-2d3d7be9a13a nodeName:}" failed. No retries permitted until 2026-07-18 02:40:42.346246778 +0000 UTC m=+65.949894442 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fvt9x" (UID: "f7b220b6-d648-4f31-b11b-2d3d7be9a13a") : configmap "kube-rbac-proxy" not found Jul 18 02:40:41.346367 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:41.346277 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-insights-runtime-extractor-tls podName:f7b220b6-d648-4f31-b11b-2d3d7be9a13a nodeName:}" failed. No retries permitted until 2026-07-18 02:40:42.3462662 +0000 UTC m=+65.949913865 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fvt9x" (UID: "f7b220b6-d648-4f31-b11b-2d3d7be9a13a") : secret "insights-runtime-extractor-tls" not found Jul 18 02:40:41.447201 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:41.447150 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/079022d6-f624-49c4-93a6-7bdb4b6a9ffd-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-scxbj\" (UID: \"079022d6-f624-49c4-93a6-7bdb4b6a9ffd\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-scxbj" Jul 18 02:40:41.447311 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:41.447288 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 18 02:40:41.447373 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:41.447350 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/079022d6-f624-49c4-93a6-7bdb4b6a9ffd-networking-console-plugin-cert podName:079022d6-f624-49c4-93a6-7bdb4b6a9ffd nodeName:}" failed. No retries permitted until 2026-07-18 02:40:49.447333483 +0000 UTC m=+73.050981167 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/079022d6-f624-49c4-93a6-7bdb4b6a9ffd-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-scxbj" (UID: "079022d6-f624-49c4-93a6-7bdb4b6a9ffd") : secret "networking-console-plugin-cert" not found Jul 18 02:40:41.448527 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:41.448496 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fadc8c8-588f-41be-94e8-4c7de1ce8649-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-hjrpw\" (UID: \"6fadc8c8-588f-41be-94e8-4c7de1ce8649\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hjrpw" Jul 18 02:40:41.451841 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:41.448704 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 18 02:40:41.451841 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:41.448763 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fadc8c8-588f-41be-94e8-4c7de1ce8649-samples-operator-tls podName:6fadc8c8-588f-41be-94e8-4c7de1ce8649 nodeName:}" failed. No retries permitted until 2026-07-18 02:40:49.448748045 +0000 UTC m=+73.052395725 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6fadc8c8-588f-41be-94e8-4c7de1ce8649-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-hjrpw" (UID: "6fadc8c8-588f-41be-94e8-4c7de1ce8649") : secret "samples-operator-tls" not found Jul 18 02:40:41.549946 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:41.549893 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f3a8a13-5e99-4e24-98ee-b112c8b3b784-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-6l557\" (UID: \"2f3a8a13-5e99-4e24-98ee-b112c8b3b784\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6l557" Jul 18 02:40:41.549946 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:41.549941 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-metrics-certs\") pod \"router-default-789b768784-4p68t\" (UID: \"7ce9775c-ae6f-4891-b9df-58d8a04ec1c8\") " pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:40:41.550094 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:41.549972 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-service-ca-bundle\") pod \"router-default-789b768784-4p68t\" (UID: \"7ce9775c-ae6f-4891-b9df-58d8a04ec1c8\") " pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:40:41.550094 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:41.550031 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 18 02:40:41.550094 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:41.550034 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 18 02:40:41.550094 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:41.550075 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-service-ca-bundle podName:7ce9775c-ae6f-4891-b9df-58d8a04ec1c8 nodeName:}" failed. No retries permitted until 2026-07-18 02:40:49.550065449 +0000 UTC m=+73.153713113 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-service-ca-bundle") pod "router-default-789b768784-4p68t" (UID: "7ce9775c-ae6f-4891-b9df-58d8a04ec1c8") : configmap references non-existent config key: service-ca.crt Jul 18 02:40:41.550094 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:41.550088 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f3a8a13-5e99-4e24-98ee-b112c8b3b784-cluster-monitoring-operator-tls podName:2f3a8a13-5e99-4e24-98ee-b112c8b3b784 nodeName:}" failed. No retries permitted until 2026-07-18 02:40:49.550083034 +0000 UTC m=+73.153730698 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2f3a8a13-5e99-4e24-98ee-b112c8b3b784-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-6l557" (UID: "2f3a8a13-5e99-4e24-98ee-b112c8b3b784") : secret "cluster-monitoring-operator-tls" not found Jul 18 02:40:41.550094 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:41.550099 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-metrics-certs podName:7ce9775c-ae6f-4891-b9df-58d8a04ec1c8 nodeName:}" failed. No retries permitted until 2026-07-18 02:40:49.550093482 +0000 UTC m=+73.153741146 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-metrics-certs") pod "router-default-789b768784-4p68t" (UID: "7ce9775c-ae6f-4891-b9df-58d8a04ec1c8") : secret "router-metrics-certs-default" not found Jul 18 02:40:42.303197 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:42.303155 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-krtsg_709575dc-6eae-43e1-9208-cc228795929a/console-operator/1.log" Jul 18 02:40:42.303546 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:42.303527 2585 scope.go:117] "RemoveContainer" containerID="9241824ff40ffcc74352a3d80d205262bdcce9d9593f0e014db64008a5bb29d1" Jul 18 02:40:42.303721 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:42.303699 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-krtsg_openshift-console-operator(709575dc-6eae-43e1-9208-cc228795929a)\"" pod="openshift-console-operator/console-operator-565b7b569d-krtsg" podUID="709575dc-6eae-43e1-9208-cc228795929a" Jul 18 02:40:42.355553 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:42.355525 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fvt9x\" (UID: \"f7b220b6-d648-4f31-b11b-2d3d7be9a13a\") " pod="openshift-insights/insights-runtime-extractor-fvt9x" Jul 18 02:40:42.355553 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:42.355555 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fvt9x\" (UID: \"f7b220b6-d648-4f31-b11b-2d3d7be9a13a\") " pod="openshift-insights/insights-runtime-extractor-fvt9x" Jul 18 02:40:42.355759 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:42.355647 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 18 02:40:42.355759 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:42.355666 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 02:40:42.355759 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:42.355711 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-kube-rbac-proxy-cm podName:f7b220b6-d648-4f31-b11b-2d3d7be9a13a nodeName:}" failed. No retries permitted until 2026-07-18 02:40:44.355694464 +0000 UTC m=+67.959342142 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fvt9x" (UID: "f7b220b6-d648-4f31-b11b-2d3d7be9a13a") : configmap "kube-rbac-proxy" not found Jul 18 02:40:42.355759 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:42.355732 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-insights-runtime-extractor-tls podName:f7b220b6-d648-4f31-b11b-2d3d7be9a13a nodeName:}" failed. No retries permitted until 2026-07-18 02:40:44.3557227 +0000 UTC m=+67.959370390 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fvt9x" (UID: "f7b220b6-d648-4f31-b11b-2d3d7be9a13a") : secret "insights-runtime-extractor-tls" not found Jul 18 02:40:42.449136 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:42.449114 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tlhsl_7321f9a8-9e03-4dff-a574-0175b0c89260/dns-node-resolver/0.log" Jul 18 02:40:42.657807 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:42.657735 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870f3a38-67e6-42ea-9398-f12eeb350f5d-metrics-certs\") pod \"network-metrics-daemon-nht5x\" (UID: \"870f3a38-67e6-42ea-9398-f12eeb350f5d\") " pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:40:42.660007 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:42.659991 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 18 02:40:42.667932 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:42.667904 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 18 02:40:42.668012 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:42.667970 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/870f3a38-67e6-42ea-9398-f12eeb350f5d-metrics-certs podName:870f3a38-67e6-42ea-9398-f12eeb350f5d nodeName:}" failed. No retries permitted until 2026-07-18 02:41:46.667952197 +0000 UTC m=+130.271599877 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/870f3a38-67e6-42ea-9398-f12eeb350f5d-metrics-certs") pod "network-metrics-daemon-nht5x" (UID: "870f3a38-67e6-42ea-9398-f12eeb350f5d") : secret "metrics-daemon-secret" not found Jul 18 02:40:42.759230 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:42.759199 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8073815-9d72-4741-8447-ca9fa5dbd049-original-pull-secret\") pod \"global-pull-secret-syncer-mqtcn\" (UID: \"d8073815-9d72-4741-8447-ca9fa5dbd049\") " pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:40:42.761659 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:42.761642 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 18 02:40:42.771841 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:42.771817 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8073815-9d72-4741-8447-ca9fa5dbd049-original-pull-secret\") pod \"global-pull-secret-syncer-mqtcn\" (UID: \"d8073815-9d72-4741-8447-ca9fa5dbd049\") " pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:40:42.860166 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:42.860141 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdl8j\" (UniqueName: \"kubernetes.io/projected/01c7795c-7254-44d2-95eb-d3f076cd4bea-kube-api-access-fdl8j\") pod \"network-check-target-8rfhc\" (UID: \"01c7795c-7254-44d2-95eb-d3f076cd4bea\") " pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:40:42.862401 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:42.862380 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdl8j\" (UniqueName: \"kubernetes.io/projected/01c7795c-7254-44d2-95eb-d3f076cd4bea-kube-api-access-fdl8j\") pod \"network-check-target-8rfhc\" (UID: \"01c7795c-7254-44d2-95eb-d3f076cd4bea\") " pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:40:42.919730 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:42.919676 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zt6zk\"" Jul 18 02:40:42.927703 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:42.927683 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:40:42.935562 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:42.935546 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqtcn" Jul 18 02:40:43.053018 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:43.052995 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-625jl_1b59997b-ff58-43bc-b393-a6c42f6f1ed4/node-ca/0.log" Jul 18 02:40:43.060273 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:43.060205 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8rfhc"] Jul 18 02:40:43.062924 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:43.062580 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-tls\") pod \"image-registry-6b7d8ff6c9-qz7qp\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:40:43.062924 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:43.062716 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e95a777-c662-4a6a-bac1-9fa079e5914c-metrics-tls\") pod \"dns-default-w4wvr\" (UID: \"7e95a777-c662-4a6a-bac1-9fa079e5914c\") " pod="openshift-dns/dns-default-w4wvr" Jul 18 02:40:43.062924 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:43.062900 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 02:40:43.063080 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:43.062960 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e95a777-c662-4a6a-bac1-9fa079e5914c-metrics-tls podName:7e95a777-c662-4a6a-bac1-9fa079e5914c nodeName:}" failed. No retries permitted until 2026-07-18 02:41:15.062941061 +0000 UTC m=+98.666588742 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e95a777-c662-4a6a-bac1-9fa079e5914c-metrics-tls") pod "dns-default-w4wvr" (UID: "7e95a777-c662-4a6a-bac1-9fa079e5914c") : secret "dns-default-metrics-tls" not found Jul 18 02:40:43.065221 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:43.063238 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 18 02:40:43.065221 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:43.063257 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp: secret "image-registry-tls" not found Jul 18 02:40:43.065221 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:43.063306 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-tls podName:b893a4ff-cd96-48b6-958f-a7ac7b736143 nodeName:}" failed. No retries permitted until 2026-07-18 02:41:15.063288461 +0000 UTC m=+98.666936142 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-tls") pod "image-registry-6b7d8ff6c9-qz7qp" (UID: "b893a4ff-cd96-48b6-958f-a7ac7b736143") : secret "image-registry-tls" not found Jul 18 02:40:43.091721 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:43.091700 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mqtcn"] Jul 18 02:40:43.093586 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:40:43.093565 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8073815_9d72_4741_8447_ca9fa5dbd049.slice/crio-9848ad2f9c1f07ca91151036ed45119b26133085b98fd683a985c763671fde88 WatchSource:0}: Error finding container 9848ad2f9c1f07ca91151036ed45119b26133085b98fd683a985c763671fde88: Status 404 returned error can't find the container with id 9848ad2f9c1f07ca91151036ed45119b26133085b98fd683a985c763671fde88 Jul 18 02:40:43.308116 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:43.308084 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8rfhc" event={"ID":"01c7795c-7254-44d2-95eb-d3f076cd4bea","Type":"ContainerStarted","Data":"90f9c64e7870d140539247b19765c33104c6cc25f05ff78665a76415a5ea52b0"} Jul 18 02:40:43.308536 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:43.308138 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8rfhc" event={"ID":"01c7795c-7254-44d2-95eb-d3f076cd4bea","Type":"ContainerStarted","Data":"f58b5c67d62234939f584e5abccb921d5f3c6b650d357b799c38f74f5a952ca3"} Jul 18 02:40:43.308536 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:43.308314 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:40:43.309311 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:43.309290 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mqtcn" event={"ID":"d8073815-9d72-4741-8447-ca9fa5dbd049","Type":"ContainerStarted","Data":"9848ad2f9c1f07ca91151036ed45119b26133085b98fd683a985c763671fde88"} Jul 18 02:40:43.323788 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:43.323751 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8rfhc" podStartSLOduration=66.323739263 podStartE2EDuration="1m6.323739263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:40:43.322651027 +0000 UTC m=+66.926298712" watchObservedRunningTime="2026-07-18 02:40:43.323739263 +0000 UTC m=+66.927386949" Jul 18 02:40:43.968510 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:43.968476 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-krtsg" Jul 18 02:40:43.968679 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:43.968534 2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-krtsg" Jul 18 02:40:43.968879 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:43.968863 2585 scope.go:117] "RemoveContainer" containerID="9241824ff40ffcc74352a3d80d205262bdcce9d9593f0e014db64008a5bb29d1" Jul 18 02:40:43.969082 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:43.969051 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-krtsg_openshift-console-operator(709575dc-6eae-43e1-9208-cc228795929a)\"" pod="openshift-console-operator/console-operator-565b7b569d-krtsg" podUID="709575dc-6eae-43e1-9208-cc228795929a" Jul 18 02:40:44.053956 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:44.053927 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vbsgm_2aceb1e5-bf1c-4344-aaa0-0e3f9fd11ab5/serve-healthcheck-canary/0.log" Jul 18 02:40:44.373701 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:44.373622 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fvt9x\" (UID: \"f7b220b6-d648-4f31-b11b-2d3d7be9a13a\") " pod="openshift-insights/insights-runtime-extractor-fvt9x" Jul 18 02:40:44.373701 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:44.373671 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fvt9x\" (UID: \"f7b220b6-d648-4f31-b11b-2d3d7be9a13a\") " pod="openshift-insights/insights-runtime-extractor-fvt9x" Jul 18 02:40:44.374165 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:44.373756 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 18 02:40:44.374165 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:44.373790 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 02:40:44.374165 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:44.373842 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-kube-rbac-proxy-cm podName:f7b220b6-d648-4f31-b11b-2d3d7be9a13a nodeName:}" failed. No retries permitted until 2026-07-18 02:40:48.373820482 +0000 UTC m=+71.977468163 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fvt9x" (UID: "f7b220b6-d648-4f31-b11b-2d3d7be9a13a") : configmap "kube-rbac-proxy" not found Jul 18 02:40:44.374165 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:44.373863 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-insights-runtime-extractor-tls podName:f7b220b6-d648-4f31-b11b-2d3d7be9a13a nodeName:}" failed. No retries permitted until 2026-07-18 02:40:48.373852132 +0000 UTC m=+71.977499796 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fvt9x" (UID: "f7b220b6-d648-4f31-b11b-2d3d7be9a13a") : secret "insights-runtime-extractor-tls" not found Jul 18 02:40:47.325376 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:47.325275 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mqtcn" event={"ID":"d8073815-9d72-4741-8447-ca9fa5dbd049","Type":"ContainerStarted","Data":"bb1d8311615a374f29979df97d9b6cd9e9ef17ca06b5ea07c7477fb60ec0d18e"} Jul 18 02:40:47.341173 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:47.341125 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mqtcn" podStartSLOduration=66.234480577 podStartE2EDuration="1m10.341112336s" totalImagesPullingTime="4.106631759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 02:39:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:40:47.340312422 +0000 UTC m=+70.943960110" watchObservedRunningTime="2026-07-18 02:40:47.341112336 +0000 UTC m=+70.944760018" Jul 18 02:40:48.411509 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:48.411478 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fvt9x\" (UID: \"f7b220b6-d648-4f31-b11b-2d3d7be9a13a\") " pod="openshift-insights/insights-runtime-extractor-fvt9x" Jul 18 02:40:48.411509 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:48.411513 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fvt9x\" (UID: \"f7b220b6-d648-4f31-b11b-2d3d7be9a13a\") " pod="openshift-insights/insights-runtime-extractor-fvt9x" Jul 18 02:40:48.411931 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:48.411609 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 02:40:48.411931 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:48.411614 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 18 02:40:48.411931 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:48.411657 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-insights-runtime-extractor-tls podName:f7b220b6-d648-4f31-b11b-2d3d7be9a13a nodeName:}" failed. No retries permitted until 2026-07-18 02:40:56.411644473 +0000 UTC m=+80.015292137 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fvt9x" (UID: "f7b220b6-d648-4f31-b11b-2d3d7be9a13a") : secret "insights-runtime-extractor-tls" not found Jul 18 02:40:48.411931 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:48.411676 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-kube-rbac-proxy-cm podName:f7b220b6-d648-4f31-b11b-2d3d7be9a13a nodeName:}" failed. No retries permitted until 2026-07-18 02:40:56.411662152 +0000 UTC m=+80.015309817 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fvt9x" (UID: "f7b220b6-d648-4f31-b11b-2d3d7be9a13a") : configmap "kube-rbac-proxy" not found Jul 18 02:40:49.519952 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:49.519923 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fadc8c8-588f-41be-94e8-4c7de1ce8649-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-hjrpw\" (UID: \"6fadc8c8-588f-41be-94e8-4c7de1ce8649\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hjrpw" Jul 18 02:40:49.520365 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:49.519963 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/079022d6-f624-49c4-93a6-7bdb4b6a9ffd-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-scxbj\" (UID: \"079022d6-f624-49c4-93a6-7bdb4b6a9ffd\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-scxbj" Jul 18 02:40:49.520365 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:49.520071 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 18 02:40:49.520365 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:49.520108 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 18 02:40:49.520365 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:49.520130 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fadc8c8-588f-41be-94e8-4c7de1ce8649-samples-operator-tls podName:6fadc8c8-588f-41be-94e8-4c7de1ce8649 nodeName:}" failed. No retries permitted until 2026-07-18 02:41:05.520115993 +0000 UTC m=+89.123763658 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6fadc8c8-588f-41be-94e8-4c7de1ce8649-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-hjrpw" (UID: "6fadc8c8-588f-41be-94e8-4c7de1ce8649") : secret "samples-operator-tls" not found Jul 18 02:40:49.520365 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:49.520157 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/079022d6-f624-49c4-93a6-7bdb4b6a9ffd-networking-console-plugin-cert podName:079022d6-f624-49c4-93a6-7bdb4b6a9ffd nodeName:}" failed. No retries permitted until 2026-07-18 02:41:05.520144677 +0000 UTC m=+89.123792342 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/079022d6-f624-49c4-93a6-7bdb4b6a9ffd-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-scxbj" (UID: "079022d6-f624-49c4-93a6-7bdb4b6a9ffd") : secret "networking-console-plugin-cert" not found Jul 18 02:40:49.620769 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:49.620742 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-metrics-certs\") pod \"router-default-789b768784-4p68t\" (UID: \"7ce9775c-ae6f-4891-b9df-58d8a04ec1c8\") " pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:40:49.620943 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:49.620781 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-service-ca-bundle\") pod \"router-default-789b768784-4p68t\" (UID: \"7ce9775c-ae6f-4891-b9df-58d8a04ec1c8\") " pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:40:49.620943 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:49.620869 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 18 02:40:49.620943 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:49.620902 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f3a8a13-5e99-4e24-98ee-b112c8b3b784-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-6l557\" (UID: \"2f3a8a13-5e99-4e24-98ee-b112c8b3b784\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6l557" Jul 18 02:40:49.620943 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:49.620916 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-metrics-certs podName:7ce9775c-ae6f-4891-b9df-58d8a04ec1c8 nodeName:}" failed. No retries permitted until 2026-07-18 02:41:05.620901237 +0000 UTC m=+89.224548918 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-metrics-certs") pod "router-default-789b768784-4p68t" (UID: "7ce9775c-ae6f-4891-b9df-58d8a04ec1c8") : secret "router-metrics-certs-default" not found Jul 18 02:40:49.621152 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:49.620954 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-service-ca-bundle podName:7ce9775c-ae6f-4891-b9df-58d8a04ec1c8 nodeName:}" failed. No retries permitted until 2026-07-18 02:41:05.620942397 +0000 UTC m=+89.224590061 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-service-ca-bundle") pod "router-default-789b768784-4p68t" (UID: "7ce9775c-ae6f-4891-b9df-58d8a04ec1c8") : configmap references non-existent config key: service-ca.crt Jul 18 02:40:49.621152 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:49.620974 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 18 02:40:49.621152 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:49.621006 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f3a8a13-5e99-4e24-98ee-b112c8b3b784-cluster-monitoring-operator-tls podName:2f3a8a13-5e99-4e24-98ee-b112c8b3b784 nodeName:}" failed. No retries permitted until 2026-07-18 02:41:05.620999018 +0000 UTC m=+89.224646682 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2f3a8a13-5e99-4e24-98ee-b112c8b3b784-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-6l557" (UID: "2f3a8a13-5e99-4e24-98ee-b112c8b3b784") : secret "cluster-monitoring-operator-tls" not found Jul 18 02:40:56.473356 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:56.473321 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fvt9x\" (UID: \"f7b220b6-d648-4f31-b11b-2d3d7be9a13a\") " pod="openshift-insights/insights-runtime-extractor-fvt9x" Jul 18 02:40:56.473356 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:56.473356 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fvt9x\" (UID: \"f7b220b6-d648-4f31-b11b-2d3d7be9a13a\") " pod="openshift-insights/insights-runtime-extractor-fvt9x" Jul 18 02:40:56.473779 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:56.473454 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 02:40:56.473779 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:40:56.473498 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-insights-runtime-extractor-tls podName:f7b220b6-d648-4f31-b11b-2d3d7be9a13a nodeName:}" failed. No retries permitted until 2026-07-18 02:41:12.473485587 +0000 UTC m=+96.077133251 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fvt9x" (UID: "f7b220b6-d648-4f31-b11b-2d3d7be9a13a") : secret "insights-runtime-extractor-tls" not found Jul 18 02:40:56.473889 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:56.473870 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fvt9x\" (UID: \"f7b220b6-d648-4f31-b11b-2d3d7be9a13a\") " pod="openshift-insights/insights-runtime-extractor-fvt9x" Jul 18 02:40:57.894841 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:57.894815 2585 scope.go:117] "RemoveContainer" containerID="9241824ff40ffcc74352a3d80d205262bdcce9d9593f0e014db64008a5bb29d1" Jul 18 02:40:58.362283 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:58.362256 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-krtsg_709575dc-6eae-43e1-9208-cc228795929a/console-operator/1.log" Jul 18 02:40:58.362432 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:58.362335 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-krtsg" event={"ID":"709575dc-6eae-43e1-9208-cc228795929a","Type":"ContainerStarted","Data":"17f55cfb2fc2bc2dee496318d252d5dd437391e8b867778b8dbece5709e95e1a"} Jul 18 02:40:58.362692 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:58.362663 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-krtsg" Jul 18 02:40:58.382559 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:58.382516 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-krtsg" podStartSLOduration=19.983104433 podStartE2EDuration="25.382481366s" totalImagesPullingTime="5.399376933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:40:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:40:58.381705092 +0000 UTC m=+81.985352778" watchObservedRunningTime="2026-07-18 02:40:58.382481366 +0000 UTC m=+81.986129052" Jul 18 02:40:58.557130 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:40:58.557101 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-krtsg" Jul 18 02:41:05.543462 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:05.543341 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fadc8c8-588f-41be-94e8-4c7de1ce8649-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-hjrpw\" (UID: \"6fadc8c8-588f-41be-94e8-4c7de1ce8649\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hjrpw" Jul 18 02:41:05.543462 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:05.543389 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/079022d6-f624-49c4-93a6-7bdb4b6a9ffd-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-scxbj\" (UID: \"079022d6-f624-49c4-93a6-7bdb4b6a9ffd\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-scxbj" Jul 18 02:41:05.545721 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:05.545700 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fadc8c8-588f-41be-94e8-4c7de1ce8649-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-hjrpw\" (UID: \"6fadc8c8-588f-41be-94e8-4c7de1ce8649\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hjrpw" Jul 18 02:41:05.545799 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:05.545709 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/079022d6-f624-49c4-93a6-7bdb4b6a9ffd-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-scxbj\" (UID: \"079022d6-f624-49c4-93a6-7bdb4b6a9ffd\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-scxbj" Jul 18 02:41:05.643926 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:05.643896 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f3a8a13-5e99-4e24-98ee-b112c8b3b784-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-6l557\" (UID: \"2f3a8a13-5e99-4e24-98ee-b112c8b3b784\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6l557" Jul 18 02:41:05.644093 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:05.643945 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-metrics-certs\") pod \"router-default-789b768784-4p68t\" (UID: \"7ce9775c-ae6f-4891-b9df-58d8a04ec1c8\") " pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:41:05.644093 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:05.643972 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-service-ca-bundle\") pod \"router-default-789b768784-4p68t\" (UID: \"7ce9775c-ae6f-4891-b9df-58d8a04ec1c8\") " pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:41:05.644539 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:05.644518 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-service-ca-bundle\") pod \"router-default-789b768784-4p68t\" (UID: \"7ce9775c-ae6f-4891-b9df-58d8a04ec1c8\") " pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:41:05.646288 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:05.646260 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2f3a8a13-5e99-4e24-98ee-b112c8b3b784-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-6l557\" (UID: \"2f3a8a13-5e99-4e24-98ee-b112c8b3b784\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6l557" Jul 18 02:41:05.646288 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:05.646260 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ce9775c-ae6f-4891-b9df-58d8a04ec1c8-metrics-certs\") pod \"router-default-789b768784-4p68t\" (UID: \"7ce9775c-ae6f-4891-b9df-58d8a04ec1c8\") " pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:41:05.757066 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:05.757042 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-6dhq7\"" Jul 18 02:41:05.764364 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:05.764343 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-jb8kd\"" Jul 18 02:41:05.765398 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:05.765380 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-scxbj" Jul 18 02:41:05.772441 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:05.772426 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hjrpw" Jul 18 02:41:05.860942 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:05.860909 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xgl4x\"" Jul 18 02:41:05.869011 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:05.868989 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6l557" Jul 18 02:41:05.871361 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:05.871296 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8lwll\"" Jul 18 02:41:05.879597 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:05.879567 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:41:05.901661 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:05.901637 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-scxbj"] Jul 18 02:41:05.904493 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:41:05.904459 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod079022d6_f624_49c4_93a6_7bdb4b6a9ffd.slice/crio-61aaa92c6ebb5dc5741ff7419a842ead4b78649034855970e64d43047f200793 WatchSource:0}: Error finding container 61aaa92c6ebb5dc5741ff7419a842ead4b78649034855970e64d43047f200793: Status 404 returned error can't find the container with id 61aaa92c6ebb5dc5741ff7419a842ead4b78649034855970e64d43047f200793 Jul 18 02:41:05.926846 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:05.926770 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hjrpw"] Jul 18 02:41:06.016838 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:06.016816 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6l557"] Jul 18 02:41:06.018988 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:41:06.018965 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f3a8a13_5e99_4e24_98ee_b112c8b3b784.slice/crio-e2552dcc7c2729ed36712b9f254a5f5cab257840a1d53e0a54c9512520cc4ce1 WatchSource:0}: Error finding container e2552dcc7c2729ed36712b9f254a5f5cab257840a1d53e0a54c9512520cc4ce1: Status 404 returned error can't find the container with id e2552dcc7c2729ed36712b9f254a5f5cab257840a1d53e0a54c9512520cc4ce1 Jul 18 02:41:06.029215 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:06.029195 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-789b768784-4p68t"] Jul 18 02:41:06.030773 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:41:06.030751 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ce9775c_ae6f_4891_b9df_58d8a04ec1c8.slice/crio-f76a30e91fd9583a6d7173300483b3719731ece7e032b304bd387ccfc5a992d2 WatchSource:0}: Error finding container f76a30e91fd9583a6d7173300483b3719731ece7e032b304bd387ccfc5a992d2: Status 404 returned error can't find the container with id f76a30e91fd9583a6d7173300483b3719731ece7e032b304bd387ccfc5a992d2 Jul 18 02:41:06.389118 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:06.389036 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-789b768784-4p68t" event={"ID":"7ce9775c-ae6f-4891-b9df-58d8a04ec1c8","Type":"ContainerStarted","Data":"0d6f6e154cd88a8a85089430bcb77e7cf5e24c613acd0f772263d4148403b836"} Jul 18 02:41:06.389118 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:06.389089 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-789b768784-4p68t" event={"ID":"7ce9775c-ae6f-4891-b9df-58d8a04ec1c8","Type":"ContainerStarted","Data":"f76a30e91fd9583a6d7173300483b3719731ece7e032b304bd387ccfc5a992d2"} Jul 18 02:41:06.390336 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:06.390306 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6l557" event={"ID":"2f3a8a13-5e99-4e24-98ee-b112c8b3b784","Type":"ContainerStarted","Data":"e2552dcc7c2729ed36712b9f254a5f5cab257840a1d53e0a54c9512520cc4ce1"} Jul 18 02:41:06.391466 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:06.391444 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-scxbj" event={"ID":"079022d6-f624-49c4-93a6-7bdb4b6a9ffd","Type":"ContainerStarted","Data":"61aaa92c6ebb5dc5741ff7419a842ead4b78649034855970e64d43047f200793"} Jul 18 02:41:06.392414 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:06.392395 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hjrpw" event={"ID":"6fadc8c8-588f-41be-94e8-4c7de1ce8649","Type":"ContainerStarted","Data":"9a26bd71dd815ebb0fb9ffb95adb068c9ce7307ea8f0e3ebf8fcf42849dc0684"} Jul 18 02:41:06.407976 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:06.407938 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-789b768784-4p68t" podStartSLOduration=33.407926186 podStartE2EDuration="33.407926186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:41:06.406468 +0000 UTC m=+90.010115686" watchObservedRunningTime="2026-07-18 02:41:06.407926186 +0000 UTC m=+90.011573871" Jul 18 02:41:06.879795 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:06.879761 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:41:06.882669 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:06.882630 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:41:07.398092 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:07.398036 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:41:07.399815 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:07.399697 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-789b768784-4p68t" Jul 18 02:41:08.402350 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:08.402308 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-scxbj" event={"ID":"079022d6-f624-49c4-93a6-7bdb4b6a9ffd","Type":"ContainerStarted","Data":"7fccf3daccf6a80506da6503f6d2cc9fbc6cdc5d5e978b85beb7d35d5ff1c2ff"} Jul 18 02:41:08.417315 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:08.417267 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-scxbj" podStartSLOduration=33.829081427 podStartE2EDuration="35.417252073s" totalImagesPullingTime="1.588170646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:40:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:41:08.416858796 +0000 UTC m=+92.020506482" watchObservedRunningTime="2026-07-18 02:41:08.417252073 +0000 UTC m=+92.020899758" Jul 18 02:41:09.408017 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:09.407982 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hjrpw" event={"ID":"6fadc8c8-588f-41be-94e8-4c7de1ce8649","Type":"ContainerStarted","Data":"dc7fd386d9fb9a9c0cd63d96ce24e34518d6f5e1c5e88d814daf4f852da2dac9"} Jul 18 02:41:09.408476 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:09.408047 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hjrpw" event={"ID":"6fadc8c8-588f-41be-94e8-4c7de1ce8649","Type":"ContainerStarted","Data":"2364070680ee6cde671ef1e1ae76d80bbe2e648a184eebf8d370b4740c13e7ff"} Jul 18 02:41:09.409603 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:09.409571 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6l557" event={"ID":"2f3a8a13-5e99-4e24-98ee-b112c8b3b784","Type":"ContainerStarted","Data":"9ca0ce313a4062adc015d9fd3e1ffc7c67576cdbfd853bf7bd9f7336688f5aa5"} Jul 18 02:41:09.424176 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:09.424126 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hjrpw" podStartSLOduration=33.393938452 podStartE2EDuration="36.424108858s" totalImagesPullingTime="3.030170406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:40:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:41:09.423470518 +0000 UTC m=+93.027118207" watchObservedRunningTime="2026-07-18 02:41:09.424108858 +0000 UTC m=+93.027756544" Jul 18 02:41:09.440562 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:09.440528 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6l557" podStartSLOduration=33.454763686 podStartE2EDuration="36.440515892s" totalImagesPullingTime="2.985752206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:40:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:41:09.439806618 +0000 UTC m=+93.043454302" watchObservedRunningTime="2026-07-18 02:41:09.440515892 +0000 UTC m=+93.044163603" Jul 18 02:41:12.505613 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:12.505575 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fvt9x\" (UID: \"f7b220b6-d648-4f31-b11b-2d3d7be9a13a\") " pod="openshift-insights/insights-runtime-extractor-fvt9x" Jul 18 02:41:12.507753 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:12.507733 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7b220b6-d648-4f31-b11b-2d3d7be9a13a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fvt9x\" (UID: \"f7b220b6-d648-4f31-b11b-2d3d7be9a13a\") " pod="openshift-insights/insights-runtime-extractor-fvt9x" Jul 18 02:41:12.782733 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:12.782666 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fvt9x" Jul 18 02:41:12.905134 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:12.905110 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fvt9x"] Jul 18 02:41:12.910292 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:41:12.910262 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b220b6_d648_4f31_b11b_2d3d7be9a13a.slice/crio-d904e4f30afe776136c6d2dd62d2091120eb07d1183b1c6838a6ced25a569f9f WatchSource:0}: Error finding container d904e4f30afe776136c6d2dd62d2091120eb07d1183b1c6838a6ced25a569f9f: Status 404 returned error can't find the container with id d904e4f30afe776136c6d2dd62d2091120eb07d1183b1c6838a6ced25a569f9f Jul 18 02:41:13.430355 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:13.430268 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fvt9x" event={"ID":"f7b220b6-d648-4f31-b11b-2d3d7be9a13a","Type":"ContainerStarted","Data":"d904e4f30afe776136c6d2dd62d2091120eb07d1183b1c6838a6ced25a569f9f"} Jul 18 02:41:14.315206 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:14.315166 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8rfhc" Jul 18 02:41:15.129789 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:15.129750 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-tls\") pod \"image-registry-6b7d8ff6c9-qz7qp\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:41:15.129965 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:15.129815 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e95a777-c662-4a6a-bac1-9fa079e5914c-metrics-tls\") pod \"dns-default-w4wvr\" (UID: \"7e95a777-c662-4a6a-bac1-9fa079e5914c\") " pod="openshift-dns/dns-default-w4wvr" Jul 18 02:41:15.132296 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:15.132272 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e95a777-c662-4a6a-bac1-9fa079e5914c-metrics-tls\") pod \"dns-default-w4wvr\" (UID: \"7e95a777-c662-4a6a-bac1-9fa079e5914c\") " pod="openshift-dns/dns-default-w4wvr" Jul 18 02:41:15.132419 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:15.132381 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-tls\") pod \"image-registry-6b7d8ff6c9-qz7qp\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:41:15.138702 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:15.138505 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7gcbm\"" Jul 18 02:41:15.145267 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:15.145250 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4pdzm\"" Jul 18 02:41:15.146280 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:15.146244 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:41:15.153760 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:15.153739 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w4wvr" Jul 18 02:41:17.039756 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.039728 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-l279g"] Jul 18 02:41:17.043403 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.043382 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l279g" Jul 18 02:41:17.046784 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.046761 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 18 02:41:17.046784 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.046760 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 18 02:41:17.046985 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.046813 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 18 02:41:17.046985 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.046875 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2x9df\"" Jul 18 02:41:17.047740 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.047726 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 18 02:41:17.149925 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.149886 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e2d82ed0-9996-45da-9757-e1226eb4dff6-node-exporter-textfile\") pod \"node-exporter-l279g\" (UID: \"e2d82ed0-9996-45da-9757-e1226eb4dff6\") " pod="openshift-monitoring/node-exporter-l279g" Jul 18 02:41:17.150083 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.149937 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2d82ed0-9996-45da-9757-e1226eb4dff6-metrics-client-ca\") pod \"node-exporter-l279g\" (UID: \"e2d82ed0-9996-45da-9757-e1226eb4dff6\") " pod="openshift-monitoring/node-exporter-l279g" Jul 18 02:41:17.150083 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.149967 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e2d82ed0-9996-45da-9757-e1226eb4dff6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l279g\" (UID: \"e2d82ed0-9996-45da-9757-e1226eb4dff6\") " pod="openshift-monitoring/node-exporter-l279g" Jul 18 02:41:17.150083 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.150002 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e2d82ed0-9996-45da-9757-e1226eb4dff6-root\") pod \"node-exporter-l279g\" (UID: \"e2d82ed0-9996-45da-9757-e1226eb4dff6\") " pod="openshift-monitoring/node-exporter-l279g" Jul 18 02:41:17.150249 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.150077 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e2d82ed0-9996-45da-9757-e1226eb4dff6-node-exporter-wtmp\") pod \"node-exporter-l279g\" (UID: \"e2d82ed0-9996-45da-9757-e1226eb4dff6\") " pod="openshift-monitoring/node-exporter-l279g" Jul 18 02:41:17.150249 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.150123 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7mmq\" (UniqueName: \"kubernetes.io/projected/e2d82ed0-9996-45da-9757-e1226eb4dff6-kube-api-access-j7mmq\") pod \"node-exporter-l279g\" (UID: \"e2d82ed0-9996-45da-9757-e1226eb4dff6\") " pod="openshift-monitoring/node-exporter-l279g" Jul 18 02:41:17.150249 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.150230 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2d82ed0-9996-45da-9757-e1226eb4dff6-sys\") pod \"node-exporter-l279g\" (UID: \"e2d82ed0-9996-45da-9757-e1226eb4dff6\") " pod="openshift-monitoring/node-exporter-l279g" Jul 18 02:41:17.150367 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.150330 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e2d82ed0-9996-45da-9757-e1226eb4dff6-node-exporter-tls\") pod \"node-exporter-l279g\" (UID: \"e2d82ed0-9996-45da-9757-e1226eb4dff6\") " pod="openshift-monitoring/node-exporter-l279g" Jul 18 02:41:17.150406 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.150365 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e2d82ed0-9996-45da-9757-e1226eb4dff6-node-exporter-accelerators-collector-config\") pod \"node-exporter-l279g\" (UID: \"e2d82ed0-9996-45da-9757-e1226eb4dff6\") " pod="openshift-monitoring/node-exporter-l279g" Jul 18 02:41:17.251231 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.251137 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2d82ed0-9996-45da-9757-e1226eb4dff6-metrics-client-ca\") pod \"node-exporter-l279g\" (UID: \"e2d82ed0-9996-45da-9757-e1226eb4dff6\") " pod="openshift-monitoring/node-exporter-l279g" Jul 18 02:41:17.251231 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.251201 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e2d82ed0-9996-45da-9757-e1226eb4dff6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l279g\" (UID: \"e2d82ed0-9996-45da-9757-e1226eb4dff6\") " pod="openshift-monitoring/node-exporter-l279g" Jul 18 02:41:17.251462 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.251242 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e2d82ed0-9996-45da-9757-e1226eb4dff6-root\") pod \"node-exporter-l279g\" (UID: \"e2d82ed0-9996-45da-9757-e1226eb4dff6\") " pod="openshift-monitoring/node-exporter-l279g" Jul 18 02:41:17.251462 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.251276 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e2d82ed0-9996-45da-9757-e1226eb4dff6-node-exporter-wtmp\") pod \"node-exporter-l279g\" (UID: \"e2d82ed0-9996-45da-9757-e1226eb4dff6\") " pod="openshift-monitoring/node-exporter-l279g" Jul 18 02:41:17.251462 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.251320 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7mmq\" (UniqueName: \"kubernetes.io/projected/e2d82ed0-9996-45da-9757-e1226eb4dff6-kube-api-access-j7mmq\") pod \"node-exporter-l279g\" (UID: \"e2d82ed0-9996-45da-9757-e1226eb4dff6\") " pod="openshift-monitoring/node-exporter-l279g" Jul 18 02:41:17.251462 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.251345 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2d82ed0-9996-45da-9757-e1226eb4dff6-sys\") pod \"node-exporter-l279g\" (UID: \"e2d82ed0-9996-45da-9757-e1226eb4dff6\") " pod="openshift-monitoring/node-exporter-l279g" Jul 18 02:41:17.251462 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.251368 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e2d82ed0-9996-45da-9757-e1226eb4dff6-root\") pod \"node-exporter-l279g\" (UID: \"e2d82ed0-9996-45da-9757-e1226eb4dff6\") " pod="openshift-monitoring/node-exporter-l279g" Jul 18 02:41:17.251709 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.251405 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e2d82ed0-9996-45da-9757-e1226eb4dff6-node-exporter-tls\") pod \"node-exporter-l279g\" (UID: \"e2d82ed0-9996-45da-9757-e1226eb4dff6\") " pod="openshift-monitoring/node-exporter-l279g" Jul 18 02:41:17.251709 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.251563 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2d82ed0-9996-45da-9757-e1226eb4dff6-sys\") pod \"node-exporter-l279g\" (UID: \"e2d82ed0-9996-45da-9757-e1226eb4dff6\") " pod="openshift-monitoring/node-exporter-l279g" Jul 18 02:41:17.251709 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.251600 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e2d82ed0-9996-45da-9757-e1226eb4dff6-node-exporter-wtmp\") pod \"node-exporter-l279g\" (UID: \"e2d82ed0-9996-45da-9757-e1226eb4dff6\") " pod="openshift-monitoring/node-exporter-l279g" Jul 18 02:41:17.251709 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.251621 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e2d82ed0-9996-45da-9757-e1226eb4dff6-node-exporter-accelerators-collector-config\") pod \"node-exporter-l279g\" (UID: \"e2d82ed0-9996-45da-9757-e1226eb4dff6\") " pod="openshift-monitoring/node-exporter-l279g" Jul 18 02:41:17.251709 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.251676 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e2d82ed0-9996-45da-9757-e1226eb4dff6-node-exporter-textfile\") pod \"node-exporter-l279g\" (UID: \"e2d82ed0-9996-45da-9757-e1226eb4dff6\") " pod="openshift-monitoring/node-exporter-l279g" Jul 18 02:41:17.251947 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.251797 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2d82ed0-9996-45da-9757-e1226eb4dff6-metrics-client-ca\") pod \"node-exporter-l279g\" (UID: \"e2d82ed0-9996-45da-9757-e1226eb4dff6\") " pod="openshift-monitoring/node-exporter-l279g" Jul 18 02:41:17.251947 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.251938 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e2d82ed0-9996-45da-9757-e1226eb4dff6-node-exporter-textfile\") pod \"node-exporter-l279g\" (UID: \"e2d82ed0-9996-45da-9757-e1226eb4dff6\") " pod="openshift-monitoring/node-exporter-l279g" Jul 18 02:41:17.252162 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.252139 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e2d82ed0-9996-45da-9757-e1226eb4dff6-node-exporter-accelerators-collector-config\") pod \"node-exporter-l279g\" (UID: \"e2d82ed0-9996-45da-9757-e1226eb4dff6\") " pod="openshift-monitoring/node-exporter-l279g" Jul 18 02:41:17.254161 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.254141 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e2d82ed0-9996-45da-9757-e1226eb4dff6-node-exporter-tls\") pod \"node-exporter-l279g\" (UID: \"e2d82ed0-9996-45da-9757-e1226eb4dff6\") " pod="openshift-monitoring/node-exporter-l279g" Jul 18 02:41:17.254291 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.254207 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e2d82ed0-9996-45da-9757-e1226eb4dff6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l279g\" (UID: \"e2d82ed0-9996-45da-9757-e1226eb4dff6\") " pod="openshift-monitoring/node-exporter-l279g" Jul 18 02:41:17.261308 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.261285 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7mmq\" (UniqueName: \"kubernetes.io/projected/e2d82ed0-9996-45da-9757-e1226eb4dff6-kube-api-access-j7mmq\") pod \"node-exporter-l279g\" (UID: \"e2d82ed0-9996-45da-9757-e1226eb4dff6\") " pod="openshift-monitoring/node-exporter-l279g" Jul 18 02:41:17.353467 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.353404 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l279g" Jul 18 02:41:17.695765 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:41:17.695736 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d82ed0_9996_45da_9757_e1226eb4dff6.slice/crio-7955dc396535b2a67a7d2a931a305f325e148bbf9238211aa85e89b5e1aa8df2 WatchSource:0}: Error finding container 7955dc396535b2a67a7d2a931a305f325e148bbf9238211aa85e89b5e1aa8df2: Status 404 returned error can't find the container with id 7955dc396535b2a67a7d2a931a305f325e148bbf9238211aa85e89b5e1aa8df2 Jul 18 02:41:17.826493 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.826472 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w4wvr"] Jul 18 02:41:17.828777 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:41:17.828754 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e95a777_c662_4a6a_bac1_9fa079e5914c.slice/crio-a819a4b6a7159dfda188409974274b6742acc1a33694e389ef863b6042751aa1 WatchSource:0}: Error finding container a819a4b6a7159dfda188409974274b6742acc1a33694e389ef863b6042751aa1: Status 404 returned error can't find the container with id a819a4b6a7159dfda188409974274b6742acc1a33694e389ef863b6042751aa1 Jul 18 02:41:17.845808 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:17.845788 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp"] Jul 18 02:41:17.847858 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:41:17.847836 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb893a4ff_cd96_48b6_958f_a7ac7b736143.slice/crio-c9db927364d786139810775b8127ac75b528bf5bafbf772e0a3a37382183d114 WatchSource:0}: Error finding container c9db927364d786139810775b8127ac75b528bf5bafbf772e0a3a37382183d114: Status 404 returned error can't find the container with id c9db927364d786139810775b8127ac75b528bf5bafbf772e0a3a37382183d114 Jul 18 02:41:18.457965 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:18.457927 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" event={"ID":"b893a4ff-cd96-48b6-958f-a7ac7b736143","Type":"ContainerStarted","Data":"b281a5727f857424f721736ff7385b8d221e4c35007c323b1434dc0b8ac5cde1"} Jul 18 02:41:18.458425 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:18.457993 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" event={"ID":"b893a4ff-cd96-48b6-958f-a7ac7b736143","Type":"ContainerStarted","Data":"c9db927364d786139810775b8127ac75b528bf5bafbf772e0a3a37382183d114"} Jul 18 02:41:18.458425 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:18.458095 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:41:18.460533 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:18.460473 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fvt9x" event={"ID":"f7b220b6-d648-4f31-b11b-2d3d7be9a13a","Type":"ContainerStarted","Data":"8ec1d665ab9475a617f4b842995b93d2301087f66089a89c040cc1994e889c6c"} Jul 18 02:41:18.462131 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:18.462101 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l279g" event={"ID":"e2d82ed0-9996-45da-9757-e1226eb4dff6","Type":"ContainerStarted","Data":"7955dc396535b2a67a7d2a931a305f325e148bbf9238211aa85e89b5e1aa8df2"} Jul 18 02:41:18.463450 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:18.463426 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w4wvr" event={"ID":"7e95a777-c662-4a6a-bac1-9fa079e5914c","Type":"ContainerStarted","Data":"a819a4b6a7159dfda188409974274b6742acc1a33694e389ef863b6042751aa1"} Jul 18 02:41:18.491139 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:18.491092 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" podStartSLOduration=101.491074727 podStartE2EDuration="1m41.491074727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:41:18.489814915 +0000 UTC m=+102.093462602" watchObservedRunningTime="2026-07-18 02:41:18.491074727 +0000 UTC m=+102.094722414" Jul 18 02:41:19.469648 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:19.469618 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l279g" event={"ID":"e2d82ed0-9996-45da-9757-e1226eb4dff6","Type":"ContainerStarted","Data":"bd9d0781076de57160e4ab33b2ad2c7632eca982b5991085e32c4f18b7f213b7"} Jul 18 02:41:20.479578 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:20.479538 2585 generic.go:356] "Generic (PLEG): container finished" podID="e2d82ed0-9996-45da-9757-e1226eb4dff6" containerID="bd9d0781076de57160e4ab33b2ad2c7632eca982b5991085e32c4f18b7f213b7" exitCode=0 Jul 18 02:41:20.479968 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:20.479615 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l279g" event={"ID":"e2d82ed0-9996-45da-9757-e1226eb4dff6","Type":"ContainerDied","Data":"bd9d0781076de57160e4ab33b2ad2c7632eca982b5991085e32c4f18b7f213b7"} Jul 18 02:41:22.489730 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:22.489660 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l279g" event={"ID":"e2d82ed0-9996-45da-9757-e1226eb4dff6","Type":"ContainerStarted","Data":"f29182184275d54dbfdcf9e67fe3a5294110324512ebabc9f73ba3db7e7cf59e"} Jul 18 02:41:22.489730 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:22.489722 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l279g" event={"ID":"e2d82ed0-9996-45da-9757-e1226eb4dff6","Type":"ContainerStarted","Data":"41ced22a237082c9065e8456e5686ad7a25760b13d74563acc33dfb03053eed5"} Jul 18 02:41:22.491938 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:22.491885 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w4wvr" event={"ID":"7e95a777-c662-4a6a-bac1-9fa079e5914c","Type":"ContainerStarted","Data":"01d7a778d2566b8f99f07476baa2fe988ffb9e38f1ac04ab5f0ee2732c562904"} Jul 18 02:41:22.491938 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:22.491936 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w4wvr" event={"ID":"7e95a777-c662-4a6a-bac1-9fa079e5914c","Type":"ContainerStarted","Data":"952d255602d4d9b0d5fef286e68edacb8f41a6e43a3983bac25b9be91f8d0bf1"} Jul 18 02:41:22.492120 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:22.491965 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w4wvr" Jul 18 02:41:22.493733 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:22.493692 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fvt9x" event={"ID":"f7b220b6-d648-4f31-b11b-2d3d7be9a13a","Type":"ContainerStarted","Data":"a8faaf4f00e3c4eaac2067b2ed6d078e82c20eefc301467f3004e0c09c3b48c5"} Jul 18 02:41:22.509793 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:22.509738 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-l279g" podStartSLOduration=3.792771615 podStartE2EDuration="5.509719428s" totalImagesPullingTime="1.622761664s" totalInitContainerRuntime="94.186149ms" isStatefulPod=true podCreationTimestamp="2026-07-18 02:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:41:22.508363883 +0000 UTC m=+106.112011563" watchObservedRunningTime="2026-07-18 02:41:22.509719428 +0000 UTC m=+106.113367120" Jul 18 02:41:22.524669 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:22.524614 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-w4wvr" podStartSLOduration=67.473192612 podStartE2EDuration="1m11.524599553s" totalImagesPullingTime="4.051406941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:40:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:41:22.523738591 +0000 UTC m=+106.127386279" watchObservedRunningTime="2026-07-18 02:41:22.524599553 +0000 UTC m=+106.128247237" Jul 18 02:41:25.509178 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:25.509145 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fvt9x" event={"ID":"f7b220b6-d648-4f31-b11b-2d3d7be9a13a","Type":"ContainerStarted","Data":"80bc92ebc71f13f5be8580e1f488678965389c7d8fe7d692e0660fe6e9f10868"} Jul 18 02:41:25.527528 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:25.527487 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fvt9x" podStartSLOduration=33.211929407 podStartE2EDuration="45.527475202s" totalImagesPullingTime="12.315545795s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 02:40:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:41:25.526439967 +0000 UTC m=+109.130087652" watchObservedRunningTime="2026-07-18 02:41:25.527475202 +0000 UTC m=+109.131122888" Jul 18 02:41:29.065101 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:29.065068 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp"] Jul 18 02:41:32.499923 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:32.499891 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w4wvr" Jul 18 02:41:39.070802 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:39.070775 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:41:41.509139 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:41.509096 2585 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" podUID="0825f880-ea1c-4683-b9b1-2e9aaeee31d8" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 18 02:41:45.582997 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:45.582966 2585 generic.go:356] "Generic (PLEG): container finished" podID="52dc3a30-b4eb-40ce-8380-9fc3a18db5df" containerID="295c34e61a24d6023ce5028ca9778112dea6dce45ed13d2748443c4e13150f90" exitCode=0 Jul 18 02:41:45.583319 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:45.583038 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-d5t6p" event={"ID":"52dc3a30-b4eb-40ce-8380-9fc3a18db5df","Type":"ContainerDied","Data":"295c34e61a24d6023ce5028ca9778112dea6dce45ed13d2748443c4e13150f90"} Jul 18 02:41:45.583416 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:45.583398 2585 scope.go:117] "RemoveContainer" containerID="295c34e61a24d6023ce5028ca9778112dea6dce45ed13d2748443c4e13150f90" Jul 18 02:41:46.229622 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:46.229592 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-789b768784-4p68t_7ce9775c-ae6f-4891-b9df-58d8a04ec1c8/router/0.log" Jul 18 02:41:46.245395 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:46.245373 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vbsgm_2aceb1e5-bf1c-4344-aaa0-0e3f9fd11ab5/serve-healthcheck-canary/0.log" Jul 18 02:41:46.588128 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:46.588062 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-d5t6p" event={"ID":"52dc3a30-b4eb-40ce-8380-9fc3a18db5df","Type":"ContainerStarted","Data":"466bcf714283fba7793dd6aa6b0b9433415a457a1b0db0e89f5367e4d5a9e7ab"} Jul 18 02:41:46.676764 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:46.676733 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870f3a38-67e6-42ea-9398-f12eeb350f5d-metrics-certs\") pod \"network-metrics-daemon-nht5x\" (UID: \"870f3a38-67e6-42ea-9398-f12eeb350f5d\") " pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:41:46.678853 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:46.678826 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870f3a38-67e6-42ea-9398-f12eeb350f5d-metrics-certs\") pod \"network-metrics-daemon-nht5x\" (UID: \"870f3a38-67e6-42ea-9398-f12eeb350f5d\") " pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:41:46.830215 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:46.830171 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hzcn2\"" Jul 18 02:41:46.838050 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:46.838029 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nht5x" Jul 18 02:41:47.005918 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:47.005893 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nht5x"] Jul 18 02:41:47.007731 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:41:47.007704 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870f3a38_67e6_42ea_9398_f12eeb350f5d.slice/crio-1c5617bcc5dbb1a40f8d757d6115c488e3aad521797430cecc90d5de60468aed WatchSource:0}: Error finding container 1c5617bcc5dbb1a40f8d757d6115c488e3aad521797430cecc90d5de60468aed: Status 404 returned error can't find the container with id 1c5617bcc5dbb1a40f8d757d6115c488e3aad521797430cecc90d5de60468aed Jul 18 02:41:47.593627 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:47.593594 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nht5x" event={"ID":"870f3a38-67e6-42ea-9398-f12eeb350f5d","Type":"ContainerStarted","Data":"1c5617bcc5dbb1a40f8d757d6115c488e3aad521797430cecc90d5de60468aed"} Jul 18 02:41:48.602351 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:48.602316 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nht5x" event={"ID":"870f3a38-67e6-42ea-9398-f12eeb350f5d","Type":"ContainerStarted","Data":"89148c8c01c833b1d0382ce9483c0e480d5923d24686ea32776e6d621aacf02f"} Jul 18 02:41:49.608516 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:49.608476 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nht5x" event={"ID":"870f3a38-67e6-42ea-9398-f12eeb350f5d","Type":"ContainerStarted","Data":"83b9886b9878dae76c785e3975ebc8136d8a5d9892638d7d583cf692d72500f5"} Jul 18 02:41:49.624410 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:49.624361 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nht5x" podStartSLOduration=131.216160868 podStartE2EDuration="2m12.624347382s" totalImagesPullingTime="1.408186514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:39:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:41:49.623307639 +0000 UTC m=+133.226955327" watchObservedRunningTime="2026-07-18 02:41:49.624347382 +0000 UTC m=+133.227995068" Jul 18 02:41:51.508914 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:51.508866 2585 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" podUID="0825f880-ea1c-4683-b9b1-2e9aaeee31d8" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 18 02:41:54.084026 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.083988 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" podUID="b893a4ff-cd96-48b6-958f-a7ac7b736143" containerName="registry" containerID="cri-o://b281a5727f857424f721736ff7385b8d221e4c35007c323b1434dc0b8ac5cde1" gracePeriod=30 Jul 18 02:41:54.317417 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.317391 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:41:54.429293 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.429224 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b893a4ff-cd96-48b6-958f-a7ac7b736143-ca-trust-extracted\") pod \"b893a4ff-cd96-48b6-958f-a7ac7b736143\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " Jul 18 02:41:54.429293 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.429267 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-tls\") pod \"b893a4ff-cd96-48b6-958f-a7ac7b736143\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " Jul 18 02:41:54.429293 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.429283 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dx299\" (UniqueName: \"kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-kube-api-access-dx299\") pod \"b893a4ff-cd96-48b6-958f-a7ac7b736143\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " Jul 18 02:41:54.429529 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.429304 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-certificates\") pod \"b893a4ff-cd96-48b6-958f-a7ac7b736143\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " Jul 18 02:41:54.429529 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.429341 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-bound-sa-token\") pod \"b893a4ff-cd96-48b6-958f-a7ac7b736143\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " Jul 18 02:41:54.429529 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.429365 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b893a4ff-cd96-48b6-958f-a7ac7b736143-image-registry-private-configuration\") pod \"b893a4ff-cd96-48b6-958f-a7ac7b736143\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " Jul 18 02:41:54.429529 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.429391 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b893a4ff-cd96-48b6-958f-a7ac7b736143-installation-pull-secrets\") pod \"b893a4ff-cd96-48b6-958f-a7ac7b736143\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " Jul 18 02:41:54.429748 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.429725 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b893a4ff-cd96-48b6-958f-a7ac7b736143-trusted-ca\") pod \"b893a4ff-cd96-48b6-958f-a7ac7b736143\" (UID: \"b893a4ff-cd96-48b6-958f-a7ac7b736143\") " Jul 18 02:41:54.430858 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.430153 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b893a4ff-cd96-48b6-958f-a7ac7b736143" (UID: "b893a4ff-cd96-48b6-958f-a7ac7b736143"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 02:41:54.430988 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.430938 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b893a4ff-cd96-48b6-958f-a7ac7b736143-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b893a4ff-cd96-48b6-958f-a7ac7b736143" (UID: "b893a4ff-cd96-48b6-958f-a7ac7b736143"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 02:41:54.432491 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.432457 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b893a4ff-cd96-48b6-958f-a7ac7b736143-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b893a4ff-cd96-48b6-958f-a7ac7b736143" (UID: "b893a4ff-cd96-48b6-958f-a7ac7b736143"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 02:41:54.433681 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.432828 2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b893a4ff-cd96-48b6-958f-a7ac7b736143-trusted-ca\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 02:41:54.433681 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.432868 2585 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-certificates\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 02:41:54.433681 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.433411 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b893a4ff-cd96-48b6-958f-a7ac7b736143" (UID: "b893a4ff-cd96-48b6-958f-a7ac7b736143"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 02:41:54.434713 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.433913 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b893a4ff-cd96-48b6-958f-a7ac7b736143" (UID: "b893a4ff-cd96-48b6-958f-a7ac7b736143"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 02:41:54.435101 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.435069 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b893a4ff-cd96-48b6-958f-a7ac7b736143-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b893a4ff-cd96-48b6-958f-a7ac7b736143" (UID: "b893a4ff-cd96-48b6-958f-a7ac7b736143"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 02:41:54.437002 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.436962 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-kube-api-access-dx299" (OuterVolumeSpecName: "kube-api-access-dx299") pod "b893a4ff-cd96-48b6-958f-a7ac7b736143" (UID: "b893a4ff-cd96-48b6-958f-a7ac7b736143"). InnerVolumeSpecName "kube-api-access-dx299". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 02:41:54.440836 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.440812 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b893a4ff-cd96-48b6-958f-a7ac7b736143-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b893a4ff-cd96-48b6-958f-a7ac7b736143" (UID: "b893a4ff-cd96-48b6-958f-a7ac7b736143"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 02:41:54.533227 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.533180 2585 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b893a4ff-cd96-48b6-958f-a7ac7b736143-installation-pull-secrets\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 02:41:54.533227 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.533225 2585 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b893a4ff-cd96-48b6-958f-a7ac7b736143-ca-trust-extracted\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 02:41:54.533349 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.533236 2585 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-registry-tls\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 02:41:54.533349 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.533245 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx299\" (UniqueName: \"kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-kube-api-access-dx299\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 02:41:54.533349 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.533253 2585 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b893a4ff-cd96-48b6-958f-a7ac7b736143-bound-sa-token\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 02:41:54.533349 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.533261 2585 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b893a4ff-cd96-48b6-958f-a7ac7b736143-image-registry-private-configuration\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 02:41:54.627495 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.627469 2585 generic.go:356] "Generic (PLEG): container finished" podID="b893a4ff-cd96-48b6-958f-a7ac7b736143" containerID="b281a5727f857424f721736ff7385b8d221e4c35007c323b1434dc0b8ac5cde1" exitCode=0 Jul 18 02:41:54.627577 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.627522 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" event={"ID":"b893a4ff-cd96-48b6-958f-a7ac7b736143","Type":"ContainerDied","Data":"b281a5727f857424f721736ff7385b8d221e4c35007c323b1434dc0b8ac5cde1"} Jul 18 02:41:54.627577 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.627532 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" Jul 18 02:41:54.627643 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.627582 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp" event={"ID":"b893a4ff-cd96-48b6-958f-a7ac7b736143","Type":"ContainerDied","Data":"c9db927364d786139810775b8127ac75b528bf5bafbf772e0a3a37382183d114"} Jul 18 02:41:54.627643 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.627601 2585 scope.go:117] "RemoveContainer" containerID="b281a5727f857424f721736ff7385b8d221e4c35007c323b1434dc0b8ac5cde1" Jul 18 02:41:54.635850 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.635836 2585 scope.go:117] "RemoveContainer" containerID="b281a5727f857424f721736ff7385b8d221e4c35007c323b1434dc0b8ac5cde1" Jul 18 02:41:54.636102 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:41:54.636084 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b281a5727f857424f721736ff7385b8d221e4c35007c323b1434dc0b8ac5cde1\": container with ID starting with b281a5727f857424f721736ff7385b8d221e4c35007c323b1434dc0b8ac5cde1 not found: ID does not exist" containerID="b281a5727f857424f721736ff7385b8d221e4c35007c323b1434dc0b8ac5cde1" Jul 18 02:41:54.636148 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.636110 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b281a5727f857424f721736ff7385b8d221e4c35007c323b1434dc0b8ac5cde1"} err="failed to get container status \"b281a5727f857424f721736ff7385b8d221e4c35007c323b1434dc0b8ac5cde1\": rpc error: code = NotFound desc = could not find container \"b281a5727f857424f721736ff7385b8d221e4c35007c323b1434dc0b8ac5cde1\": container with ID starting with b281a5727f857424f721736ff7385b8d221e4c35007c323b1434dc0b8ac5cde1 not found: ID does not exist" Jul 18 02:41:54.648332 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.648310 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp"] Jul 18 02:41:54.652091 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.652067 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6b7d8ff6c9-qz7qp"] Jul 18 02:41:54.899586 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:41:54.899560 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b893a4ff-cd96-48b6-958f-a7ac7b736143" path="/var/lib/kubelet/pods/b893a4ff-cd96-48b6-958f-a7ac7b736143/volumes" Jul 18 02:42:00.654002 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:42:00.653965 2585 generic.go:356] "Generic (PLEG): container finished" podID="53d15949-a9fb-4897-bec5-feb443eb99d9" containerID="5f231e47886c80536b57cdd11dccee55a3c44756819165654a03d98778be9e51" exitCode=0 Jul 18 02:42:00.654420 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:42:00.654011 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-gchp4" event={"ID":"53d15949-a9fb-4897-bec5-feb443eb99d9","Type":"ContainerDied","Data":"5f231e47886c80536b57cdd11dccee55a3c44756819165654a03d98778be9e51"} Jul 18 02:42:00.654420 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:42:00.654345 2585 scope.go:117] "RemoveContainer" containerID="5f231e47886c80536b57cdd11dccee55a3c44756819165654a03d98778be9e51" Jul 18 02:42:01.508687 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:42:01.508649 2585 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" podUID="0825f880-ea1c-4683-b9b1-2e9aaeee31d8" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 18 02:42:01.508832 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:42:01.508721 2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" Jul 18 02:42:01.509210 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:42:01.509171 2585 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"bd137bcb93a9d988a172dee503af082e8b027209927135ca1106e6c48f5297d2"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 18 02:42:01.509251 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:42:01.509230 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" podUID="0825f880-ea1c-4683-b9b1-2e9aaeee31d8" containerName="service-proxy" containerID="cri-o://bd137bcb93a9d988a172dee503af082e8b027209927135ca1106e6c48f5297d2" gracePeriod=30 Jul 18 02:42:01.660330 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:42:01.660303 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-gchp4" event={"ID":"53d15949-a9fb-4897-bec5-feb443eb99d9","Type":"ContainerStarted","Data":"1e77493a3ca9df89e974925d6ad17c173313e1bc8d3f3900599c9c6510f6cfe8"} Jul 18 02:42:01.662643 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:42:01.662555 2585 generic.go:356] "Generic (PLEG): container finished" podID="0825f880-ea1c-4683-b9b1-2e9aaeee31d8" containerID="bd137bcb93a9d988a172dee503af082e8b027209927135ca1106e6c48f5297d2" exitCode=2 Jul 18 02:42:01.662643 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:42:01.662635 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" event={"ID":"0825f880-ea1c-4683-b9b1-2e9aaeee31d8","Type":"ContainerDied","Data":"bd137bcb93a9d988a172dee503af082e8b027209927135ca1106e6c48f5297d2"} Jul 18 02:42:02.668009 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:42:02.667972 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8f6f455-bvsp8" event={"ID":"0825f880-ea1c-4683-b9b1-2e9aaeee31d8","Type":"ContainerStarted","Data":"f8aa108d6cd11b9c95888b057e34e7a3f96d2d72aad32ccc8ea724eb14535c9b"} Jul 18 02:42:10.698008 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:42:10.697963 2585 generic.go:356] "Generic (PLEG): container finished" podID="e05161ba-27ca-4679-9439-bad0efa4500f" containerID="2f5e2ce5a278080654560df39b2942932e44412124bf9d24142362a24a1c84e2" exitCode=0 Jul 18 02:42:10.698398 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:42:10.698032 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-c4l7k" event={"ID":"e05161ba-27ca-4679-9439-bad0efa4500f","Type":"ContainerDied","Data":"2f5e2ce5a278080654560df39b2942932e44412124bf9d24142362a24a1c84e2"} Jul 18 02:42:10.698398 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:42:10.698380 2585 scope.go:117] "RemoveContainer" containerID="2f5e2ce5a278080654560df39b2942932e44412124bf9d24142362a24a1c84e2" Jul 18 02:42:11.704217 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:42:11.704165 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-c4l7k" event={"ID":"e05161ba-27ca-4679-9439-bad0efa4500f","Type":"ContainerStarted","Data":"9e7e7a3538d35d99931d3ce6f4ee81255cf3c85f54b9d63f2d9ec725c9148214"} Jul 18 02:44:24.061692 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:24.061655 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f9896c96c-2v8km"] Jul 18 02:44:24.062117 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:24.061932 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b893a4ff-cd96-48b6-958f-a7ac7b736143" containerName="registry" Jul 18 02:44:24.062117 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:24.061942 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b893a4ff-cd96-48b6-958f-a7ac7b736143" containerName="registry" Jul 18 02:44:24.062117 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:24.061987 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b893a4ff-cd96-48b6-958f-a7ac7b736143" containerName="registry" Jul 18 02:44:24.064846 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:24.064829 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f9896c96c-2v8km" Jul 18 02:44:24.068103 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:24.068082 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 18 02:44:24.068103 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:24.068090 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 18 02:44:24.068304 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:24.068179 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 18 02:44:24.069202 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:24.069163 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 18 02:44:24.071048 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:24.070788 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-gxrnd\"" Jul 18 02:44:24.117290 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:24.117265 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f9896c96c-2v8km"] Jul 18 02:44:24.167424 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:24.167392 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf965887-ba5b-4fef-808f-0006359dd779-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f9896c96c-2v8km\" (UID: \"cf965887-ba5b-4fef-808f-0006359dd779\") " pod="opendatahub/opendatahub-operator-controller-manager-5f9896c96c-2v8km" Jul 18 02:44:24.167571 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:24.167520 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf965887-ba5b-4fef-808f-0006359dd779-tmp\") pod \"opendatahub-operator-controller-manager-5f9896c96c-2v8km\" (UID: \"cf965887-ba5b-4fef-808f-0006359dd779\") " pod="opendatahub/opendatahub-operator-controller-manager-5f9896c96c-2v8km" Jul 18 02:44:24.167630 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:24.167581 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf965887-ba5b-4fef-808f-0006359dd779-manifests\") pod \"opendatahub-operator-controller-manager-5f9896c96c-2v8km\" (UID: \"cf965887-ba5b-4fef-808f-0006359dd779\") " pod="opendatahub/opendatahub-operator-controller-manager-5f9896c96c-2v8km" Jul 18 02:44:24.167630 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:24.167616 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptkmm\" (UniqueName: \"kubernetes.io/projected/cf965887-ba5b-4fef-808f-0006359dd779-kube-api-access-ptkmm\") pod \"opendatahub-operator-controller-manager-5f9896c96c-2v8km\" (UID: \"cf965887-ba5b-4fef-808f-0006359dd779\") " pod="opendatahub/opendatahub-operator-controller-manager-5f9896c96c-2v8km" Jul 18 02:44:24.167725 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:24.167668 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf965887-ba5b-4fef-808f-0006359dd779-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f9896c96c-2v8km\" (UID: \"cf965887-ba5b-4fef-808f-0006359dd779\") " pod="opendatahub/opendatahub-operator-controller-manager-5f9896c96c-2v8km" Jul 18 02:44:24.268278 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:24.268249 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf965887-ba5b-4fef-808f-0006359dd779-tmp\") pod \"opendatahub-operator-controller-manager-5f9896c96c-2v8km\" (UID: \"cf965887-ba5b-4fef-808f-0006359dd779\") " pod="opendatahub/opendatahub-operator-controller-manager-5f9896c96c-2v8km" Jul 18 02:44:24.268425 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:24.268287 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf965887-ba5b-4fef-808f-0006359dd779-manifests\") pod \"opendatahub-operator-controller-manager-5f9896c96c-2v8km\" (UID: \"cf965887-ba5b-4fef-808f-0006359dd779\") " pod="opendatahub/opendatahub-operator-controller-manager-5f9896c96c-2v8km" Jul 18 02:44:24.268425 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:24.268312 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptkmm\" (UniqueName: \"kubernetes.io/projected/cf965887-ba5b-4fef-808f-0006359dd779-kube-api-access-ptkmm\") pod \"opendatahub-operator-controller-manager-5f9896c96c-2v8km\" (UID: \"cf965887-ba5b-4fef-808f-0006359dd779\") " pod="opendatahub/opendatahub-operator-controller-manager-5f9896c96c-2v8km" Jul 18 02:44:24.268425 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:24.268335 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf965887-ba5b-4fef-808f-0006359dd779-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f9896c96c-2v8km\" (UID: \"cf965887-ba5b-4fef-808f-0006359dd779\") " pod="opendatahub/opendatahub-operator-controller-manager-5f9896c96c-2v8km" Jul 18 02:44:24.268425 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:24.268400 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf965887-ba5b-4fef-808f-0006359dd779-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f9896c96c-2v8km\" (UID: \"cf965887-ba5b-4fef-808f-0006359dd779\") " pod="opendatahub/opendatahub-operator-controller-manager-5f9896c96c-2v8km" Jul 18 02:44:24.268699 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:24.268673 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf965887-ba5b-4fef-808f-0006359dd779-manifests\") pod \"opendatahub-operator-controller-manager-5f9896c96c-2v8km\" (UID: \"cf965887-ba5b-4fef-808f-0006359dd779\") " pod="opendatahub/opendatahub-operator-controller-manager-5f9896c96c-2v8km" Jul 18 02:44:24.270565 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:24.270537 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf965887-ba5b-4fef-808f-0006359dd779-tmp\") pod \"opendatahub-operator-controller-manager-5f9896c96c-2v8km\" (UID: \"cf965887-ba5b-4fef-808f-0006359dd779\") " pod="opendatahub/opendatahub-operator-controller-manager-5f9896c96c-2v8km" Jul 18 02:44:24.270655 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:24.270629 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf965887-ba5b-4fef-808f-0006359dd779-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f9896c96c-2v8km\" (UID: \"cf965887-ba5b-4fef-808f-0006359dd779\") " pod="opendatahub/opendatahub-operator-controller-manager-5f9896c96c-2v8km" Jul 18 02:44:24.270694 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:24.270661 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf965887-ba5b-4fef-808f-0006359dd779-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f9896c96c-2v8km\" (UID: \"cf965887-ba5b-4fef-808f-0006359dd779\") " pod="opendatahub/opendatahub-operator-controller-manager-5f9896c96c-2v8km" Jul 18 02:44:24.278660 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:24.278638 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptkmm\" (UniqueName: \"kubernetes.io/projected/cf965887-ba5b-4fef-808f-0006359dd779-kube-api-access-ptkmm\") pod \"opendatahub-operator-controller-manager-5f9896c96c-2v8km\" (UID: \"cf965887-ba5b-4fef-808f-0006359dd779\") " pod="opendatahub/opendatahub-operator-controller-manager-5f9896c96c-2v8km" Jul 18 02:44:24.374035 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:24.373981 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f9896c96c-2v8km" Jul 18 02:44:24.500279 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:24.500253 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f9896c96c-2v8km"] Jul 18 02:44:24.502690 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:44:24.502661 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf965887_ba5b_4fef_808f_0006359dd779.slice/crio-f84fd03a7ef98c914108ddc15328e10e78a9de03559a8f9ab86cdd9c16b32e2a WatchSource:0}: Error finding container f84fd03a7ef98c914108ddc15328e10e78a9de03559a8f9ab86cdd9c16b32e2a: Status 404 returned error can't find the container with id f84fd03a7ef98c914108ddc15328e10e78a9de03559a8f9ab86cdd9c16b32e2a Jul 18 02:44:25.174031 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:25.173989 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f9896c96c-2v8km" event={"ID":"cf965887-ba5b-4fef-808f-0006359dd779","Type":"ContainerStarted","Data":"f84fd03a7ef98c914108ddc15328e10e78a9de03559a8f9ab86cdd9c16b32e2a"} Jul 18 02:44:28.188519 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:28.188484 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f9896c96c-2v8km" event={"ID":"cf965887-ba5b-4fef-808f-0006359dd779","Type":"ContainerStarted","Data":"5ae6eac06dcfafaac40153cb416d1d0ceb7a039137a742227e5eb1d24e104bab"} Jul 18 02:44:29.194088 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:29.194056 2585 generic.go:356] "Generic (PLEG): container finished" podID="cf965887-ba5b-4fef-808f-0006359dd779" containerID="5ae6eac06dcfafaac40153cb416d1d0ceb7a039137a742227e5eb1d24e104bab" exitCode=0 Jul 18 02:44:29.194488 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:29.194095 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f9896c96c-2v8km" event={"ID":"cf965887-ba5b-4fef-808f-0006359dd779","Type":"ContainerDied","Data":"5ae6eac06dcfafaac40153cb416d1d0ceb7a039137a742227e5eb1d24e104bab"} Jul 18 02:44:30.200401 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:30.200365 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f9896c96c-2v8km" event={"ID":"cf965887-ba5b-4fef-808f-0006359dd779","Type":"ContainerStarted","Data":"e3c3b7547287acbe0065a13510fc6e1c11c184137412a9d93a5f708e532b91fe"} Jul 18 02:44:30.200876 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:30.200500 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5f9896c96c-2v8km" Jul 18 02:44:30.222977 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:30.222926 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5f9896c96c-2v8km" podStartSLOduration=2.089603794 podStartE2EDuration="6.222914449s" totalImagesPullingTime="3.137470228s" totalInitContainerRuntime="995.840427ms" isStatefulPod=false podCreationTimestamp="2026-07-18 02:44:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:44:30.221033633 +0000 UTC m=+293.824681320" watchObservedRunningTime="2026-07-18 02:44:30.222914449 +0000 UTC m=+293.826562134" Jul 18 02:44:36.815538 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:36.815510 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-krtsg_709575dc-6eae-43e1-9208-cc228795929a/console-operator/1.log" Jul 18 02:44:36.819542 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:36.819517 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-krtsg_709575dc-6eae-43e1-9208-cc228795929a/console-operator/1.log" Jul 18 02:44:36.828771 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:36.828745 2585 kubelet.go:1648] "Image garbage collection succeeded" Jul 18 02:44:41.206893 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:41.206862 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5f9896c96c-2v8km" Jul 18 02:44:43.567913 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:43.567880 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-58dcbd754f-vt7hf"] Jul 18 02:44:43.571801 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:43.571774 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58dcbd754f-vt7hf" Jul 18 02:44:43.574881 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:43.574859 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 18 02:44:43.575356 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:43.575336 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 18 02:44:43.575945 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:43.575929 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-kb2fh\"" Jul 18 02:44:43.592102 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:43.592079 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58dcbd754f-vt7hf"] Jul 18 02:44:43.714130 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:43.714088 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69eed15a-f1c7-4677-b76a-8aceb91dae02-tls-certs\") pod \"kube-auth-proxy-58dcbd754f-vt7hf\" (UID: \"69eed15a-f1c7-4677-b76a-8aceb91dae02\") " pod="openshift-ingress/kube-auth-proxy-58dcbd754f-vt7hf" Jul 18 02:44:43.714321 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:43.714213 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxmv5\" (UniqueName: \"kubernetes.io/projected/69eed15a-f1c7-4677-b76a-8aceb91dae02-kube-api-access-hxmv5\") pod \"kube-auth-proxy-58dcbd754f-vt7hf\" (UID: \"69eed15a-f1c7-4677-b76a-8aceb91dae02\") " pod="openshift-ingress/kube-auth-proxy-58dcbd754f-vt7hf" Jul 18 02:44:43.714321 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:43.714264 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69eed15a-f1c7-4677-b76a-8aceb91dae02-tmp\") pod \"kube-auth-proxy-58dcbd754f-vt7hf\" (UID: \"69eed15a-f1c7-4677-b76a-8aceb91dae02\") " pod="openshift-ingress/kube-auth-proxy-58dcbd754f-vt7hf" Jul 18 02:44:43.815596 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:43.815558 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxmv5\" (UniqueName: \"kubernetes.io/projected/69eed15a-f1c7-4677-b76a-8aceb91dae02-kube-api-access-hxmv5\") pod \"kube-auth-proxy-58dcbd754f-vt7hf\" (UID: \"69eed15a-f1c7-4677-b76a-8aceb91dae02\") " pod="openshift-ingress/kube-auth-proxy-58dcbd754f-vt7hf" Jul 18 02:44:43.815596 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:43.815600 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69eed15a-f1c7-4677-b76a-8aceb91dae02-tmp\") pod \"kube-auth-proxy-58dcbd754f-vt7hf\" (UID: \"69eed15a-f1c7-4677-b76a-8aceb91dae02\") " pod="openshift-ingress/kube-auth-proxy-58dcbd754f-vt7hf" Jul 18 02:44:43.815845 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:43.815641 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69eed15a-f1c7-4677-b76a-8aceb91dae02-tls-certs\") pod \"kube-auth-proxy-58dcbd754f-vt7hf\" (UID: \"69eed15a-f1c7-4677-b76a-8aceb91dae02\") " pod="openshift-ingress/kube-auth-proxy-58dcbd754f-vt7hf" Jul 18 02:44:43.815845 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:44:43.815761 2585 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 18 02:44:43.815845 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:44:43.815825 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69eed15a-f1c7-4677-b76a-8aceb91dae02-tls-certs podName:69eed15a-f1c7-4677-b76a-8aceb91dae02 nodeName:}" failed. No retries permitted until 2026-07-18 02:44:44.315805259 +0000 UTC m=+307.919452939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/69eed15a-f1c7-4677-b76a-8aceb91dae02-tls-certs") pod "kube-auth-proxy-58dcbd754f-vt7hf" (UID: "69eed15a-f1c7-4677-b76a-8aceb91dae02") : secret "kube-auth-proxy-tls" not found Jul 18 02:44:43.817822 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:43.817801 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69eed15a-f1c7-4677-b76a-8aceb91dae02-tmp\") pod \"kube-auth-proxy-58dcbd754f-vt7hf\" (UID: \"69eed15a-f1c7-4677-b76a-8aceb91dae02\") " pod="openshift-ingress/kube-auth-proxy-58dcbd754f-vt7hf" Jul 18 02:44:43.825209 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:43.825135 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxmv5\" (UniqueName: \"kubernetes.io/projected/69eed15a-f1c7-4677-b76a-8aceb91dae02-kube-api-access-hxmv5\") pod \"kube-auth-proxy-58dcbd754f-vt7hf\" (UID: \"69eed15a-f1c7-4677-b76a-8aceb91dae02\") " pod="openshift-ingress/kube-auth-proxy-58dcbd754f-vt7hf" Jul 18 02:44:43.903834 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:43.903806 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-qw7nf"] Jul 18 02:44:43.907224 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:43.907179 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-qw7nf" Jul 18 02:44:43.910446 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:43.910420 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-8hpf8\"" Jul 18 02:44:43.910536 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:43.910518 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 18 02:44:43.920570 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:43.920544 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-qw7nf"] Jul 18 02:44:44.017523 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:44.017489 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29bc75bf-af7f-4462-ae2a-df308ef86ec9-cert\") pod \"odh-model-controller-7998c67dc4-qw7nf\" (UID: \"29bc75bf-af7f-4462-ae2a-df308ef86ec9\") " pod="opendatahub/odh-model-controller-7998c67dc4-qw7nf" Jul 18 02:44:44.017681 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:44.017528 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr4d9\" (UniqueName: \"kubernetes.io/projected/29bc75bf-af7f-4462-ae2a-df308ef86ec9-kube-api-access-qr4d9\") pod \"odh-model-controller-7998c67dc4-qw7nf\" (UID: \"29bc75bf-af7f-4462-ae2a-df308ef86ec9\") " pod="opendatahub/odh-model-controller-7998c67dc4-qw7nf" Jul 18 02:44:44.118233 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:44.118139 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29bc75bf-af7f-4462-ae2a-df308ef86ec9-cert\") pod \"odh-model-controller-7998c67dc4-qw7nf\" (UID: \"29bc75bf-af7f-4462-ae2a-df308ef86ec9\") " pod="opendatahub/odh-model-controller-7998c67dc4-qw7nf" Jul 18 02:44:44.118233 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:44.118209 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr4d9\" (UniqueName: \"kubernetes.io/projected/29bc75bf-af7f-4462-ae2a-df308ef86ec9-kube-api-access-qr4d9\") pod \"odh-model-controller-7998c67dc4-qw7nf\" (UID: \"29bc75bf-af7f-4462-ae2a-df308ef86ec9\") " pod="opendatahub/odh-model-controller-7998c67dc4-qw7nf" Jul 18 02:44:44.118403 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:44:44.118290 2585 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 18 02:44:44.118403 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:44:44.118375 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29bc75bf-af7f-4462-ae2a-df308ef86ec9-cert podName:29bc75bf-af7f-4462-ae2a-df308ef86ec9 nodeName:}" failed. No retries permitted until 2026-07-18 02:44:44.618358222 +0000 UTC m=+308.222005899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/29bc75bf-af7f-4462-ae2a-df308ef86ec9-cert") pod "odh-model-controller-7998c67dc4-qw7nf" (UID: "29bc75bf-af7f-4462-ae2a-df308ef86ec9") : secret "odh-model-controller-webhook-cert" not found Jul 18 02:44:44.127088 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:44.127063 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr4d9\" (UniqueName: \"kubernetes.io/projected/29bc75bf-af7f-4462-ae2a-df308ef86ec9-kube-api-access-qr4d9\") pod \"odh-model-controller-7998c67dc4-qw7nf\" (UID: \"29bc75bf-af7f-4462-ae2a-df308ef86ec9\") " pod="opendatahub/odh-model-controller-7998c67dc4-qw7nf" Jul 18 02:44:44.320337 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:44.320309 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69eed15a-f1c7-4677-b76a-8aceb91dae02-tls-certs\") pod \"kube-auth-proxy-58dcbd754f-vt7hf\" (UID: \"69eed15a-f1c7-4677-b76a-8aceb91dae02\") " pod="openshift-ingress/kube-auth-proxy-58dcbd754f-vt7hf" Jul 18 02:44:44.322625 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:44.322598 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69eed15a-f1c7-4677-b76a-8aceb91dae02-tls-certs\") pod \"kube-auth-proxy-58dcbd754f-vt7hf\" (UID: \"69eed15a-f1c7-4677-b76a-8aceb91dae02\") " pod="openshift-ingress/kube-auth-proxy-58dcbd754f-vt7hf" Jul 18 02:44:44.482951 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:44.482917 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58dcbd754f-vt7hf" Jul 18 02:44:44.605813 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:44.605678 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58dcbd754f-vt7hf"] Jul 18 02:44:44.608512 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:44:44.608486 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69eed15a_f1c7_4677_b76a_8aceb91dae02.slice/crio-c8ac1c5eaedaacb8684ad5ef1bf927bda579f8040ab1837acca5bd1e6fa68d59 WatchSource:0}: Error finding container c8ac1c5eaedaacb8684ad5ef1bf927bda579f8040ab1837acca5bd1e6fa68d59: Status 404 returned error can't find the container with id c8ac1c5eaedaacb8684ad5ef1bf927bda579f8040ab1837acca5bd1e6fa68d59 Jul 18 02:44:44.610324 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:44.610306 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 02:44:44.621823 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:44.621804 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29bc75bf-af7f-4462-ae2a-df308ef86ec9-cert\") pod \"odh-model-controller-7998c67dc4-qw7nf\" (UID: \"29bc75bf-af7f-4462-ae2a-df308ef86ec9\") " pod="opendatahub/odh-model-controller-7998c67dc4-qw7nf" Jul 18 02:44:44.623984 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:44.623967 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29bc75bf-af7f-4462-ae2a-df308ef86ec9-cert\") pod \"odh-model-controller-7998c67dc4-qw7nf\" (UID: \"29bc75bf-af7f-4462-ae2a-df308ef86ec9\") " pod="opendatahub/odh-model-controller-7998c67dc4-qw7nf" Jul 18 02:44:44.759985 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:44.759913 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc"] Jul 18 02:44:44.764364 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:44.764342 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" Jul 18 02:44:44.766933 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:44.766914 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 18 02:44:44.767026 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:44.766914 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 18 02:44:44.767213 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:44.767198 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8h859\"" Jul 18 02:44:44.767341 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:44.767323 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 18 02:44:44.767395 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:44.767338 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 18 02:44:44.775917 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:44.775894 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc"] Jul 18 02:44:44.817260 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:44.817235 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-qw7nf" Jul 18 02:44:44.924120 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:44.924081 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9b830f58-3f21-4d70-9b91-ac32ca48e29f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-9wnjc\" (UID: \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" Jul 18 02:44:44.924120 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:44.924120 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b830f58-3f21-4d70-9b91-ac32ca48e29f-local-certs\") pod \"istiod-openshift-gateway-7574459656-9wnjc\" (UID: \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" Jul 18 02:44:44.924337 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:44.924160 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9b830f58-3f21-4d70-9b91-ac32ca48e29f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-9wnjc\" (UID: \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" Jul 18 02:44:44.924337 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:44.924218 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9b830f58-3f21-4d70-9b91-ac32ca48e29f-cacerts\") pod \"istiod-openshift-gateway-7574459656-9wnjc\" (UID: \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" Jul 18 02:44:44.924337 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:44.924240 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b830f58-3f21-4d70-9b91-ac32ca48e29f-istio-token\") pod \"istiod-openshift-gateway-7574459656-9wnjc\" (UID: \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" Jul 18 02:44:44.924337 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:44.924265 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9b830f58-3f21-4d70-9b91-ac32ca48e29f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-9wnjc\" (UID: \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" Jul 18 02:44:44.924460 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:44.924395 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t8zw\" (UniqueName: \"kubernetes.io/projected/9b830f58-3f21-4d70-9b91-ac32ca48e29f-kube-api-access-5t8zw\") pod \"istiod-openshift-gateway-7574459656-9wnjc\" (UID: \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" Jul 18 02:44:44.972364 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:44.972335 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-qw7nf"] Jul 18 02:44:45.025799 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:45.025728 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9b830f58-3f21-4d70-9b91-ac32ca48e29f-cacerts\") pod \"istiod-openshift-gateway-7574459656-9wnjc\" (UID: \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" Jul 18 02:44:45.025799 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:45.025765 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b830f58-3f21-4d70-9b91-ac32ca48e29f-istio-token\") pod \"istiod-openshift-gateway-7574459656-9wnjc\" (UID: \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" Jul 18 02:44:45.025799 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:45.025792 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9b830f58-3f21-4d70-9b91-ac32ca48e29f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-9wnjc\" (UID: \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" Jul 18 02:44:45.026013 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:45.025845 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t8zw\" (UniqueName: \"kubernetes.io/projected/9b830f58-3f21-4d70-9b91-ac32ca48e29f-kube-api-access-5t8zw\") pod \"istiod-openshift-gateway-7574459656-9wnjc\" (UID: \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" Jul 18 02:44:45.026013 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:45.025863 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9b830f58-3f21-4d70-9b91-ac32ca48e29f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-9wnjc\" (UID: \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" Jul 18 02:44:45.026013 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:45.025895 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b830f58-3f21-4d70-9b91-ac32ca48e29f-local-certs\") pod \"istiod-openshift-gateway-7574459656-9wnjc\" (UID: \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" Jul 18 02:44:45.026013 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:45.025948 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9b830f58-3f21-4d70-9b91-ac32ca48e29f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-9wnjc\" (UID: \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" Jul 18 02:44:45.026701 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:45.026676 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9b830f58-3f21-4d70-9b91-ac32ca48e29f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-9wnjc\" (UID: \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" Jul 18 02:44:45.028376 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:45.028353 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9b830f58-3f21-4d70-9b91-ac32ca48e29f-cacerts\") pod \"istiod-openshift-gateway-7574459656-9wnjc\" (UID: \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" Jul 18 02:44:45.028376 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:45.028364 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b830f58-3f21-4d70-9b91-ac32ca48e29f-local-certs\") pod \"istiod-openshift-gateway-7574459656-9wnjc\" (UID: \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" Jul 18 02:44:45.028739 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:45.028721 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9b830f58-3f21-4d70-9b91-ac32ca48e29f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-9wnjc\" (UID: \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" Jul 18 02:44:45.028854 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:45.028835 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9b830f58-3f21-4d70-9b91-ac32ca48e29f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-9wnjc\" (UID: \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" Jul 18 02:44:45.033915 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:45.033885 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b830f58-3f21-4d70-9b91-ac32ca48e29f-istio-token\") pod \"istiod-openshift-gateway-7574459656-9wnjc\" (UID: \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" Jul 18 02:44:45.034314 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:45.034297 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t8zw\" (UniqueName: \"kubernetes.io/projected/9b830f58-3f21-4d70-9b91-ac32ca48e29f-kube-api-access-5t8zw\") pod \"istiod-openshift-gateway-7574459656-9wnjc\" (UID: \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" Jul 18 02:44:45.073394 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:45.073373 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" Jul 18 02:44:45.214029 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:45.213995 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc"] Jul 18 02:44:45.216487 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:44:45.216461 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b830f58_3f21_4d70_9b91_ac32ca48e29f.slice/crio-421f462d551c1d9fbcbedcc5ac9a109e04371f00347b87aa142b9efa11e0a402 WatchSource:0}: Error finding container 421f462d551c1d9fbcbedcc5ac9a109e04371f00347b87aa142b9efa11e0a402: Status 404 returned error can't find the container with id 421f462d551c1d9fbcbedcc5ac9a109e04371f00347b87aa142b9efa11e0a402 Jul 18 02:44:45.262917 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:45.262877 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" event={"ID":"9b830f58-3f21-4d70-9b91-ac32ca48e29f","Type":"ContainerStarted","Data":"421f462d551c1d9fbcbedcc5ac9a109e04371f00347b87aa142b9efa11e0a402"} Jul 18 02:44:45.264106 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:45.264075 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-qw7nf" event={"ID":"29bc75bf-af7f-4462-ae2a-df308ef86ec9","Type":"ContainerStarted","Data":"5c9afa5aded4849210a8ded955062873b3da03bae1e4068aa1040dd4bf8c6ca6"} Jul 18 02:44:45.265339 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:45.265317 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58dcbd754f-vt7hf" event={"ID":"69eed15a-f1c7-4677-b76a-8aceb91dae02","Type":"ContainerStarted","Data":"c8ac1c5eaedaacb8684ad5ef1bf927bda579f8040ab1837acca5bd1e6fa68d59"} Jul 18 02:44:49.338316 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:49.338273 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 18 02:44:49.338635 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:49.338347 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 18 02:44:50.296089 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:50.296050 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-qw7nf" event={"ID":"29bc75bf-af7f-4462-ae2a-df308ef86ec9","Type":"ContainerStarted","Data":"64df490ebf421bcf9e08b52887dfb299e6ecd602dd0c91430b075154676436da"} Jul 18 02:44:50.296310 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:50.296163 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-qw7nf" Jul 18 02:44:50.297630 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:50.297605 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58dcbd754f-vt7hf" event={"ID":"69eed15a-f1c7-4677-b76a-8aceb91dae02","Type":"ContainerStarted","Data":"5eddaaec850eba507a8d773e873cc062fd8322e9c779c7a5f0f6fea0a403c530"} Jul 18 02:44:50.299369 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:50.299347 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" event={"ID":"9b830f58-3f21-4d70-9b91-ac32ca48e29f","Type":"ContainerStarted","Data":"a6b5dd0d6cec9c01b16e33063b1d439a9e1298b791cd7b11c1b403b29e67ba47"} Jul 18 02:44:50.299470 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:50.299385 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" Jul 18 02:44:50.301070 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:50.301047 2585 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-9wnjc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 18 02:44:50.301153 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:50.301098 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" podUID="9b830f58-3f21-4d70-9b91-ac32ca48e29f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 18 02:44:50.314681 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:50.314634 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-qw7nf" podStartSLOduration=2.9423869209999998 podStartE2EDuration="7.31461745s" totalImagesPullingTime="4.372230529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:44:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:44:50.312196253 +0000 UTC m=+313.915843931" watchObservedRunningTime="2026-07-18 02:44:50.31461745 +0000 UTC m=+313.918265137" Jul 18 02:44:50.330790 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:50.330744 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-58dcbd754f-vt7hf" podStartSLOduration=2.60554632 podStartE2EDuration="7.330728236s" totalImagesPullingTime="4.725181916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:44:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:44:50.329371176 +0000 UTC m=+313.933018865" watchObservedRunningTime="2026-07-18 02:44:50.330728236 +0000 UTC m=+313.934375923" Jul 18 02:44:50.351808 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:50.351762 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" podStartSLOduration=2.232408156 podStartE2EDuration="6.351745387s" totalImagesPullingTime="4.119337231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:44:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:44:50.349605408 +0000 UTC m=+313.953253095" watchObservedRunningTime="2026-07-18 02:44:50.351745387 +0000 UTC m=+313.955393074" Jul 18 02:44:51.078445 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:51.078414 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qjp49"] Jul 18 02:44:51.081687 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:51.081668 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qjp49" Jul 18 02:44:51.116946 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:51.116923 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qjp49"] Jul 18 02:44:51.181129 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:51.181053 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-qjp49\" (UID: \"0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qjp49" Jul 18 02:44:51.181129 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:51.181108 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-qjp49\" (UID: \"0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qjp49" Jul 18 02:44:51.181356 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:51.181165 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-qjp49\" (UID: \"0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qjp49" Jul 18 02:44:51.181356 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:51.181236 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-qjp49\" (UID: \"0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qjp49" Jul 18 02:44:51.181356 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:51.181271 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-qjp49\" (UID: \"0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qjp49" Jul 18 02:44:51.181356 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:51.181347 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-qjp49\" (UID: \"0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qjp49" Jul 18 02:44:51.181570 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:51.181380 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmgzc\" (UniqueName: \"kubernetes.io/projected/0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98-kube-api-access-gmgzc\") pod \"istiod-openshift-gateway-94bb8fbfd-qjp49\" (UID: \"0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qjp49" Jul 18 02:44:51.281851 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:51.281821 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-qjp49\" (UID: \"0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qjp49" Jul 18 02:44:51.282073 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:51.281861 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-qjp49\" (UID: \"0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qjp49" Jul 18 02:44:51.282073 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:51.281896 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-qjp49\" (UID: \"0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qjp49" Jul 18 02:44:51.282073 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:51.281912 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-qjp49\" (UID: \"0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qjp49" Jul 18 02:44:51.282073 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:51.281936 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-qjp49\" (UID: \"0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qjp49" Jul 18 02:44:51.282073 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:51.281960 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-qjp49\" (UID: \"0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qjp49" Jul 18 02:44:51.282073 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:51.281979 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmgzc\" (UniqueName: \"kubernetes.io/projected/0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98-kube-api-access-gmgzc\") pod \"istiod-openshift-gateway-94bb8fbfd-qjp49\" (UID: \"0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qjp49" Jul 18 02:44:51.282651 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:51.282615 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-qjp49\" (UID: \"0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qjp49" Jul 18 02:44:51.284073 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:51.284052 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-qjp49\" (UID: \"0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qjp49" Jul 18 02:44:51.284345 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:51.284323 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-qjp49\" (UID: \"0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qjp49" Jul 18 02:44:51.284398 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:51.284333 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-qjp49\" (UID: \"0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qjp49" Jul 18 02:44:51.284432 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:51.284402 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-qjp49\" (UID: \"0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qjp49" Jul 18 02:44:51.295002 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:51.294983 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-qjp49\" (UID: \"0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qjp49" Jul 18 02:44:51.295433 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:51.295406 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmgzc\" (UniqueName: \"kubernetes.io/projected/0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98-kube-api-access-gmgzc\") pod \"istiod-openshift-gateway-94bb8fbfd-qjp49\" (UID: \"0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qjp49" Jul 18 02:44:51.304605 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:51.304583 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" Jul 18 02:44:51.391286 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:51.391215 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qjp49" Jul 18 02:44:51.546856 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:51.546831 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qjp49"] Jul 18 02:44:51.558682 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:51.558652 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 18 02:44:51.558780 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:51.558707 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 18 02:44:52.309370 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:52.309340 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qjp49" event={"ID":"0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98","Type":"ContainerStarted","Data":"df63bb309e157d2617d3354d74d15021b26819eae7853bb54feb9d194ad0b721"} Jul 18 02:44:52.309514 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:52.309381 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qjp49" event={"ID":"0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98","Type":"ContainerStarted","Data":"4bf777d69f66d1b14fab7889b5fe19658e0cedff130f94cbafca1e5baf4450d3"} Jul 18 02:44:52.342271 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:52.342222 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qjp49" podStartSLOduration=1.342206936 podStartE2EDuration="1.342206936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:44:52.339738477 +0000 UTC m=+315.943386162" watchObservedRunningTime="2026-07-18 02:44:52.342206936 +0000 UTC m=+315.945854619" Jul 18 02:44:53.314012 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.313968 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qjp49" Jul 18 02:44:53.315665 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.315643 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qjp49" Jul 18 02:44:53.382509 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.382483 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc"] Jul 18 02:44:53.382652 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.382626 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" podUID="9b830f58-3f21-4d70-9b91-ac32ca48e29f" containerName="discovery" containerID="cri-o://a6b5dd0d6cec9c01b16e33063b1d439a9e1298b791cd7b11c1b403b29e67ba47" gracePeriod=30 Jul 18 02:44:53.624430 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.624407 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" Jul 18 02:44:53.804261 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.804232 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9b830f58-3f21-4d70-9b91-ac32ca48e29f-cacerts\") pod \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\" (UID: \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\") " Jul 18 02:44:53.804414 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.804283 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9b830f58-3f21-4d70-9b91-ac32ca48e29f-istio-csr-ca-configmap\") pod \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\" (UID: \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\") " Jul 18 02:44:53.804414 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.804347 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9b830f58-3f21-4d70-9b91-ac32ca48e29f-istio-csr-dns-cert\") pod \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\" (UID: \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\") " Jul 18 02:44:53.804414 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.804376 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b830f58-3f21-4d70-9b91-ac32ca48e29f-istio-token\") pod \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\" (UID: \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\") " Jul 18 02:44:53.804565 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.804504 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9b830f58-3f21-4d70-9b91-ac32ca48e29f-istio-kubeconfig\") pod \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\" (UID: \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\") " Jul 18 02:44:53.804565 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.804546 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b830f58-3f21-4d70-9b91-ac32ca48e29f-local-certs\") pod \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\" (UID: \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\") " Jul 18 02:44:53.804667 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.804605 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5t8zw\" (UniqueName: \"kubernetes.io/projected/9b830f58-3f21-4d70-9b91-ac32ca48e29f-kube-api-access-5t8zw\") pod \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\" (UID: \"9b830f58-3f21-4d70-9b91-ac32ca48e29f\") " Jul 18 02:44:53.804759 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.804730 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b830f58-3f21-4d70-9b91-ac32ca48e29f-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "9b830f58-3f21-4d70-9b91-ac32ca48e29f" (UID: "9b830f58-3f21-4d70-9b91-ac32ca48e29f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 02:44:53.806684 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.806659 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b830f58-3f21-4d70-9b91-ac32ca48e29f-cacerts" (OuterVolumeSpecName: "cacerts") pod "9b830f58-3f21-4d70-9b91-ac32ca48e29f" (UID: "9b830f58-3f21-4d70-9b91-ac32ca48e29f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 02:44:53.806849 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.806782 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b830f58-3f21-4d70-9b91-ac32ca48e29f-istio-token" (OuterVolumeSpecName: "istio-token") pod "9b830f58-3f21-4d70-9b91-ac32ca48e29f" (UID: "9b830f58-3f21-4d70-9b91-ac32ca48e29f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 02:44:53.807011 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.806990 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b830f58-3f21-4d70-9b91-ac32ca48e29f-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "9b830f58-3f21-4d70-9b91-ac32ca48e29f" (UID: "9b830f58-3f21-4d70-9b91-ac32ca48e29f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 02:44:53.807071 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.807004 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b830f58-3f21-4d70-9b91-ac32ca48e29f-local-certs" (OuterVolumeSpecName: "local-certs") pod "9b830f58-3f21-4d70-9b91-ac32ca48e29f" (UID: "9b830f58-3f21-4d70-9b91-ac32ca48e29f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 02:44:53.807170 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.807149 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b830f58-3f21-4d70-9b91-ac32ca48e29f-kube-api-access-5t8zw" (OuterVolumeSpecName: "kube-api-access-5t8zw") pod "9b830f58-3f21-4d70-9b91-ac32ca48e29f" (UID: "9b830f58-3f21-4d70-9b91-ac32ca48e29f"). InnerVolumeSpecName "kube-api-access-5t8zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 02:44:53.807229 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.807158 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b830f58-3f21-4d70-9b91-ac32ca48e29f-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "9b830f58-3f21-4d70-9b91-ac32ca48e29f" (UID: "9b830f58-3f21-4d70-9b91-ac32ca48e29f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 02:44:53.906096 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.906028 2585 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9b830f58-3f21-4d70-9b91-ac32ca48e29f-cacerts\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 02:44:53.906096 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.906056 2585 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9b830f58-3f21-4d70-9b91-ac32ca48e29f-istio-csr-ca-configmap\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 02:44:53.906096 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.906070 2585 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9b830f58-3f21-4d70-9b91-ac32ca48e29f-istio-csr-dns-cert\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 02:44:53.906096 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.906083 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b830f58-3f21-4d70-9b91-ac32ca48e29f-istio-token\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 02:44:53.906096 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.906095 2585 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9b830f58-3f21-4d70-9b91-ac32ca48e29f-istio-kubeconfig\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 02:44:53.906374 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.906106 2585 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b830f58-3f21-4d70-9b91-ac32ca48e29f-local-certs\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 02:44:53.906374 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.906116 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t8zw\" (UniqueName: \"kubernetes.io/projected/9b830f58-3f21-4d70-9b91-ac32ca48e29f-kube-api-access-5t8zw\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 02:44:53.983977 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.983939 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-t7wfl"] Jul 18 02:44:53.984332 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.984314 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b830f58-3f21-4d70-9b91-ac32ca48e29f" containerName="discovery" Jul 18 02:44:53.984422 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.984334 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b830f58-3f21-4d70-9b91-ac32ca48e29f" containerName="discovery" Jul 18 02:44:53.984422 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.984402 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b830f58-3f21-4d70-9b91-ac32ca48e29f" containerName="discovery" Jul 18 02:44:53.987639 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.987620 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t7wfl" Jul 18 02:44:53.990322 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.990302 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-v58lx\"" Jul 18 02:44:53.990404 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:53.990349 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 18 02:44:54.002868 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:54.002847 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-t7wfl"] Jul 18 02:44:54.108050 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:54.108020 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef50dd81-64dc-4c75-968c-68794de8d2d3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-t7wfl\" (UID: \"ef50dd81-64dc-4c75-968c-68794de8d2d3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t7wfl" Jul 18 02:44:54.108050 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:54.108052 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grvdf\" (UniqueName: \"kubernetes.io/projected/ef50dd81-64dc-4c75-968c-68794de8d2d3-kube-api-access-grvdf\") pod \"llmisvc-controller-manager-777c6bd7d5-t7wfl\" (UID: \"ef50dd81-64dc-4c75-968c-68794de8d2d3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t7wfl" Jul 18 02:44:54.209387 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:54.209302 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef50dd81-64dc-4c75-968c-68794de8d2d3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-t7wfl\" (UID: \"ef50dd81-64dc-4c75-968c-68794de8d2d3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t7wfl" Jul 18 02:44:54.209387 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:54.209339 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grvdf\" (UniqueName: \"kubernetes.io/projected/ef50dd81-64dc-4c75-968c-68794de8d2d3-kube-api-access-grvdf\") pod \"llmisvc-controller-manager-777c6bd7d5-t7wfl\" (UID: \"ef50dd81-64dc-4c75-968c-68794de8d2d3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t7wfl" Jul 18 02:44:54.211651 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:54.211631 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef50dd81-64dc-4c75-968c-68794de8d2d3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-t7wfl\" (UID: \"ef50dd81-64dc-4c75-968c-68794de8d2d3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t7wfl" Jul 18 02:44:54.221397 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:54.221377 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grvdf\" (UniqueName: \"kubernetes.io/projected/ef50dd81-64dc-4c75-968c-68794de8d2d3-kube-api-access-grvdf\") pod \"llmisvc-controller-manager-777c6bd7d5-t7wfl\" (UID: \"ef50dd81-64dc-4c75-968c-68794de8d2d3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t7wfl" Jul 18 02:44:54.297104 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:54.297081 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t7wfl" Jul 18 02:44:54.319718 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:54.319690 2585 generic.go:356] "Generic (PLEG): container finished" podID="9b830f58-3f21-4d70-9b91-ac32ca48e29f" containerID="a6b5dd0d6cec9c01b16e33063b1d439a9e1298b791cd7b11c1b403b29e67ba47" exitCode=0 Jul 18 02:44:54.320064 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:54.319729 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" Jul 18 02:44:54.320064 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:54.319831 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" event={"ID":"9b830f58-3f21-4d70-9b91-ac32ca48e29f","Type":"ContainerDied","Data":"a6b5dd0d6cec9c01b16e33063b1d439a9e1298b791cd7b11c1b403b29e67ba47"} Jul 18 02:44:54.320064 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:54.319941 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc" event={"ID":"9b830f58-3f21-4d70-9b91-ac32ca48e29f","Type":"ContainerDied","Data":"421f462d551c1d9fbcbedcc5ac9a109e04371f00347b87aa142b9efa11e0a402"} Jul 18 02:44:54.320064 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:54.319980 2585 scope.go:117] "RemoveContainer" containerID="a6b5dd0d6cec9c01b16e33063b1d439a9e1298b791cd7b11c1b403b29e67ba47" Jul 18 02:44:54.328476 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:54.328454 2585 scope.go:117] "RemoveContainer" containerID="a6b5dd0d6cec9c01b16e33063b1d439a9e1298b791cd7b11c1b403b29e67ba47" Jul 18 02:44:54.328746 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:44:54.328728 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6b5dd0d6cec9c01b16e33063b1d439a9e1298b791cd7b11c1b403b29e67ba47\": container with ID starting with a6b5dd0d6cec9c01b16e33063b1d439a9e1298b791cd7b11c1b403b29e67ba47 not found: ID does not exist" containerID="a6b5dd0d6cec9c01b16e33063b1d439a9e1298b791cd7b11c1b403b29e67ba47" Jul 18 02:44:54.328811 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:54.328758 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6b5dd0d6cec9c01b16e33063b1d439a9e1298b791cd7b11c1b403b29e67ba47"} err="failed to get container status \"a6b5dd0d6cec9c01b16e33063b1d439a9e1298b791cd7b11c1b403b29e67ba47\": rpc error: code = NotFound desc = could not find container \"a6b5dd0d6cec9c01b16e33063b1d439a9e1298b791cd7b11c1b403b29e67ba47\": container with ID starting with a6b5dd0d6cec9c01b16e33063b1d439a9e1298b791cd7b11c1b403b29e67ba47 not found: ID does not exist" Jul 18 02:44:54.344460 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:54.344435 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc"] Jul 18 02:44:54.349875 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:54.349850 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9wnjc"] Jul 18 02:44:54.423887 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:54.423865 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-t7wfl"] Jul 18 02:44:54.425824 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:44:54.425801 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podef50dd81_64dc_4c75_968c_68794de8d2d3.slice/crio-169685f45ab175807c4aac439dc9d0bf4a803fc412c9a95b8205786ca754a81b WatchSource:0}: Error finding container 169685f45ab175807c4aac439dc9d0bf4a803fc412c9a95b8205786ca754a81b: Status 404 returned error can't find the container with id 169685f45ab175807c4aac439dc9d0bf4a803fc412c9a95b8205786ca754a81b Jul 18 02:44:54.900400 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:54.900370 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b830f58-3f21-4d70-9b91-ac32ca48e29f" path="/var/lib/kubelet/pods/9b830f58-3f21-4d70-9b91-ac32ca48e29f/volumes" Jul 18 02:44:55.331844 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:55.331809 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t7wfl" event={"ID":"ef50dd81-64dc-4c75-968c-68794de8d2d3","Type":"ContainerStarted","Data":"169685f45ab175807c4aac439dc9d0bf4a803fc412c9a95b8205786ca754a81b"} Jul 18 02:44:57.341614 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:57.341536 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t7wfl" event={"ID":"ef50dd81-64dc-4c75-968c-68794de8d2d3","Type":"ContainerStarted","Data":"d270c8d60ee0bd1c550c6e684394b9d623493ccc7b40ac5b6e82fe910f5a10e1"} Jul 18 02:44:57.341929 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:57.341672 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t7wfl" Jul 18 02:44:57.360405 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:44:57.360361 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t7wfl" podStartSLOduration=1.784303527 podStartE2EDuration="4.360346459s" totalImagesPullingTime="2.576042932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:44:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:44:57.358327974 +0000 UTC m=+320.961975659" watchObservedRunningTime="2026-07-18 02:44:57.360346459 +0000 UTC m=+320.963994189" Jul 18 02:45:01.306592 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:45:01.306555 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-qw7nf" Jul 18 02:45:28.348266 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:45:28.348230 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t7wfl" Jul 18 02:46:10.662138 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:10.662100 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xbd9k"] Jul 18 02:46:10.665142 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:10.665120 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xbd9k" Jul 18 02:46:10.669302 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:10.669279 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 18 02:46:10.669486 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:10.669473 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 18 02:46:10.670665 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:10.670648 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-s22xd\"" Jul 18 02:46:10.688111 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:10.688090 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xbd9k"] Jul 18 02:46:10.757576 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:10.757552 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf8lc\" (UniqueName: \"kubernetes.io/projected/4a816ec7-9910-4e2b-a3f1-f71f23996ce2-kube-api-access-gf8lc\") pod \"limitador-operator-controller-manager-859d7b78bd-xbd9k\" (UID: \"4a816ec7-9910-4e2b-a3f1-f71f23996ce2\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xbd9k" Jul 18 02:46:10.858407 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:10.858382 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf8lc\" (UniqueName: \"kubernetes.io/projected/4a816ec7-9910-4e2b-a3f1-f71f23996ce2-kube-api-access-gf8lc\") pod \"limitador-operator-controller-manager-859d7b78bd-xbd9k\" (UID: \"4a816ec7-9910-4e2b-a3f1-f71f23996ce2\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xbd9k" Jul 18 02:46:10.882878 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:10.882849 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf8lc\" (UniqueName: \"kubernetes.io/projected/4a816ec7-9910-4e2b-a3f1-f71f23996ce2-kube-api-access-gf8lc\") pod \"limitador-operator-controller-manager-859d7b78bd-xbd9k\" (UID: \"4a816ec7-9910-4e2b-a3f1-f71f23996ce2\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xbd9k" Jul 18 02:46:10.975164 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:10.975141 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xbd9k" Jul 18 02:46:11.113026 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:11.112997 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xbd9k"] Jul 18 02:46:11.114610 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:46:11.114585 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a816ec7_9910_4e2b_a3f1_f71f23996ce2.slice/crio-bf4b61581314e5fb30ebfcc630108a4d05c3c7160290f3cd7fc98a4152195b85 WatchSource:0}: Error finding container bf4b61581314e5fb30ebfcc630108a4d05c3c7160290f3cd7fc98a4152195b85: Status 404 returned error can't find the container with id bf4b61581314e5fb30ebfcc630108a4d05c3c7160290f3cd7fc98a4152195b85 Jul 18 02:46:11.656836 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:11.656799 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xbd9k" event={"ID":"4a816ec7-9910-4e2b-a3f1-f71f23996ce2","Type":"ContainerStarted","Data":"bf4b61581314e5fb30ebfcc630108a4d05c3c7160290f3cd7fc98a4152195b85"} Jul 18 02:46:13.670325 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:13.670288 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xbd9k" event={"ID":"4a816ec7-9910-4e2b-a3f1-f71f23996ce2","Type":"ContainerStarted","Data":"9ab1f35c7c85fdd5f577e0b57b3101bf443ac4f242a6908a06f2f46175a54beb"} Jul 18 02:46:13.670711 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:13.670391 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xbd9k" Jul 18 02:46:13.704459 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:13.704409 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xbd9k" podStartSLOduration=1.9969952439999998 podStartE2EDuration="3.704395298s" totalImagesPullingTime="1.707400054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:46:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:46:13.702106401 +0000 UTC m=+397.305754087" watchObservedRunningTime="2026-07-18 02:46:13.704395298 +0000 UTC m=+397.308042983" Jul 18 02:46:23.106985 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:23.106953 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-fhgtx"] Jul 18 02:46:23.110265 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:23.110247 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fhgtx" Jul 18 02:46:23.113144 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:23.113122 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-98spx\"" Jul 18 02:46:23.113486 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:23.113467 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 18 02:46:23.129737 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:23.129717 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-fhgtx"] Jul 18 02:46:23.258364 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:23.258339 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgq69\" (UniqueName: \"kubernetes.io/projected/a3c5af49-8a96-4792-8441-f44c1aa8aca7-kube-api-access-jgq69\") pod \"dns-operator-controller-manager-864dd78c4c-fhgtx\" (UID: \"a3c5af49-8a96-4792-8441-f44c1aa8aca7\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fhgtx" Jul 18 02:46:23.359420 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:23.359358 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgq69\" (UniqueName: \"kubernetes.io/projected/a3c5af49-8a96-4792-8441-f44c1aa8aca7-kube-api-access-jgq69\") pod \"dns-operator-controller-manager-864dd78c4c-fhgtx\" (UID: \"a3c5af49-8a96-4792-8441-f44c1aa8aca7\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fhgtx" Jul 18 02:46:23.369579 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:23.369551 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgq69\" (UniqueName: \"kubernetes.io/projected/a3c5af49-8a96-4792-8441-f44c1aa8aca7-kube-api-access-jgq69\") pod \"dns-operator-controller-manager-864dd78c4c-fhgtx\" (UID: \"a3c5af49-8a96-4792-8441-f44c1aa8aca7\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fhgtx" Jul 18 02:46:23.420404 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:23.420384 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fhgtx" Jul 18 02:46:23.545787 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:23.545761 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-fhgtx"] Jul 18 02:46:23.547864 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:46:23.547838 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c5af49_8a96_4792_8441_f44c1aa8aca7.slice/crio-7d239dc6799ad3744af3e2c25e4cefaab02d6bc47f8f989f7ae925f56d5a86ec WatchSource:0}: Error finding container 7d239dc6799ad3744af3e2c25e4cefaab02d6bc47f8f989f7ae925f56d5a86ec: Status 404 returned error can't find the container with id 7d239dc6799ad3744af3e2c25e4cefaab02d6bc47f8f989f7ae925f56d5a86ec Jul 18 02:46:23.714473 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:23.714411 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fhgtx" event={"ID":"a3c5af49-8a96-4792-8441-f44c1aa8aca7","Type":"ContainerStarted","Data":"7d239dc6799ad3744af3e2c25e4cefaab02d6bc47f8f989f7ae925f56d5a86ec"} Jul 18 02:46:24.676946 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:24.676918 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xbd9k" Jul 18 02:46:25.725887 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:25.725853 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fhgtx" event={"ID":"a3c5af49-8a96-4792-8441-f44c1aa8aca7","Type":"ContainerStarted","Data":"87d0af41a07fb532e38e057a8c82aef48d10bf0c235deab4085f0de131754c8d"} Jul 18 02:46:25.726317 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:25.725930 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fhgtx" Jul 18 02:46:25.750577 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:25.750530 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fhgtx" podStartSLOduration=1.067623903 podStartE2EDuration="2.750515447s" totalImagesPullingTime="1.682891544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:46:25.74938465 +0000 UTC m=+409.353032335" watchObservedRunningTime="2026-07-18 02:46:25.750515447 +0000 UTC m=+409.354163133" Jul 18 02:46:35.800312 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:35.800277 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xbd9k"] Jul 18 02:46:35.800709 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:35.800471 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xbd9k" podUID="4a816ec7-9910-4e2b-a3f1-f71f23996ce2" containerName="manager" containerID="cri-o://9ab1f35c7c85fdd5f577e0b57b3101bf443ac4f242a6908a06f2f46175a54beb" gracePeriod=2 Jul 18 02:46:35.825007 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:46:35.824963 2585 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"4a816ec7-9910-4e2b-a3f1-f71f23996ce2\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-xbd9k\": pods \"limitador-operator-controller-manager-859d7b78bd-xbd9k\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xbd9k" Jul 18 02:46:35.830199 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:35.826712 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xbd9k"] Jul 18 02:46:35.906150 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:35.906115 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-65jzn"] Jul 18 02:46:35.906498 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:35.906485 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a816ec7-9910-4e2b-a3f1-f71f23996ce2" containerName="manager" Jul 18 02:46:35.906549 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:35.906500 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a816ec7-9910-4e2b-a3f1-f71f23996ce2" containerName="manager" Jul 18 02:46:35.906582 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:35.906570 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a816ec7-9910-4e2b-a3f1-f71f23996ce2" containerName="manager" Jul 18 02:46:35.909632 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:35.909608 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-65jzn" Jul 18 02:46:35.913575 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:46:35.913548 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xbd9k\" is forbidden: User \"system:node:ip-10-0-140-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-4.ec2.internal' and this object" podUID="4a816ec7-9910-4e2b-a3f1-f71f23996ce2" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xbd9k" Jul 18 02:46:35.914118 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:35.914100 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-mtc7r\"" Jul 18 02:46:35.943548 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:35.943519 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-65jzn"] Jul 18 02:46:35.952615 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:35.952583 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/198c17a7-93e7-4928-8a23-751025f59e4d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-65jzn\" (UID: \"198c17a7-93e7-4928-8a23-751025f59e4d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-65jzn" Jul 18 02:46:35.952753 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:35.952729 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pslk\" (UniqueName: \"kubernetes.io/projected/198c17a7-93e7-4928-8a23-751025f59e4d-kube-api-access-9pslk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-65jzn\" (UID: \"198c17a7-93e7-4928-8a23-751025f59e4d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-65jzn" Jul 18 02:46:36.042074 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:36.042052 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xbd9k" Jul 18 02:46:36.045069 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:46:36.045046 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xbd9k\" is forbidden: User \"system:node:ip-10-0-140-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-4.ec2.internal' and this object" podUID="4a816ec7-9910-4e2b-a3f1-f71f23996ce2" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xbd9k" Jul 18 02:46:36.053410 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:36.053349 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gf8lc\" (UniqueName: \"kubernetes.io/projected/4a816ec7-9910-4e2b-a3f1-f71f23996ce2-kube-api-access-gf8lc\") pod \"4a816ec7-9910-4e2b-a3f1-f71f23996ce2\" (UID: \"4a816ec7-9910-4e2b-a3f1-f71f23996ce2\") " Jul 18 02:46:36.053526 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:36.053505 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pslk\" (UniqueName: \"kubernetes.io/projected/198c17a7-93e7-4928-8a23-751025f59e4d-kube-api-access-9pslk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-65jzn\" (UID: \"198c17a7-93e7-4928-8a23-751025f59e4d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-65jzn" Jul 18 02:46:36.053593 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:36.053560 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/198c17a7-93e7-4928-8a23-751025f59e4d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-65jzn\" (UID: \"198c17a7-93e7-4928-8a23-751025f59e4d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-65jzn" Jul 18 02:46:36.053917 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:36.053899 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/198c17a7-93e7-4928-8a23-751025f59e4d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-65jzn\" (UID: \"198c17a7-93e7-4928-8a23-751025f59e4d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-65jzn" Jul 18 02:46:36.055386 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:36.055356 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a816ec7-9910-4e2b-a3f1-f71f23996ce2-kube-api-access-gf8lc" (OuterVolumeSpecName: "kube-api-access-gf8lc") pod "4a816ec7-9910-4e2b-a3f1-f71f23996ce2" (UID: "4a816ec7-9910-4e2b-a3f1-f71f23996ce2"). InnerVolumeSpecName "kube-api-access-gf8lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 02:46:36.076542 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:36.076520 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pslk\" (UniqueName: \"kubernetes.io/projected/198c17a7-93e7-4928-8a23-751025f59e4d-kube-api-access-9pslk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-65jzn\" (UID: \"198c17a7-93e7-4928-8a23-751025f59e4d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-65jzn" Jul 18 02:46:36.154654 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:36.154631 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf8lc\" (UniqueName: \"kubernetes.io/projected/4a816ec7-9910-4e2b-a3f1-f71f23996ce2-kube-api-access-gf8lc\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 02:46:36.220272 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:36.220242 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-65jzn" Jul 18 02:46:36.396274 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:36.396248 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-65jzn"] Jul 18 02:46:36.398018 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:46:36.397991 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod198c17a7_93e7_4928_8a23_751025f59e4d.slice/crio-d0aa56023b308b0d44667494c15f1f23f3de69640b318c71235546892158f01c WatchSource:0}: Error finding container d0aa56023b308b0d44667494c15f1f23f3de69640b318c71235546892158f01c: Status 404 returned error can't find the container with id d0aa56023b308b0d44667494c15f1f23f3de69640b318c71235546892158f01c Jul 18 02:46:36.732314 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:36.732293 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fhgtx" Jul 18 02:46:36.736805 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:46:36.736780 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xbd9k\" is forbidden: User \"system:node:ip-10-0-140-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-4.ec2.internal' and this object" podUID="4a816ec7-9910-4e2b-a3f1-f71f23996ce2" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xbd9k" Jul 18 02:46:36.776392 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:36.776368 2585 generic.go:356] "Generic (PLEG): container finished" podID="4a816ec7-9910-4e2b-a3f1-f71f23996ce2" containerID="9ab1f35c7c85fdd5f577e0b57b3101bf443ac4f242a6908a06f2f46175a54beb" exitCode=0 Jul 18 02:46:36.776506 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:36.776406 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xbd9k" Jul 18 02:46:36.776506 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:36.776442 2585 scope.go:117] "RemoveContainer" containerID="9ab1f35c7c85fdd5f577e0b57b3101bf443ac4f242a6908a06f2f46175a54beb" Jul 18 02:46:36.778093 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:36.778048 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-65jzn" event={"ID":"198c17a7-93e7-4928-8a23-751025f59e4d","Type":"ContainerStarted","Data":"d0aa56023b308b0d44667494c15f1f23f3de69640b318c71235546892158f01c"} Jul 18 02:46:36.784136 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:46:36.784103 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xbd9k\" is forbidden: User \"system:node:ip-10-0-140-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-4.ec2.internal' and this object" podUID="4a816ec7-9910-4e2b-a3f1-f71f23996ce2" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xbd9k" Jul 18 02:46:36.785002 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:36.784983 2585 scope.go:117] "RemoveContainer" containerID="9ab1f35c7c85fdd5f577e0b57b3101bf443ac4f242a6908a06f2f46175a54beb" Jul 18 02:46:36.785325 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:46:36.785305 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ab1f35c7c85fdd5f577e0b57b3101bf443ac4f242a6908a06f2f46175a54beb\": container with ID starting with 9ab1f35c7c85fdd5f577e0b57b3101bf443ac4f242a6908a06f2f46175a54beb not found: ID does not exist" containerID="9ab1f35c7c85fdd5f577e0b57b3101bf443ac4f242a6908a06f2f46175a54beb" Jul 18 02:46:36.785416 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:36.785331 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab1f35c7c85fdd5f577e0b57b3101bf443ac4f242a6908a06f2f46175a54beb"} err="failed to get container status \"9ab1f35c7c85fdd5f577e0b57b3101bf443ac4f242a6908a06f2f46175a54beb\": rpc error: code = NotFound desc = could not find container \"9ab1f35c7c85fdd5f577e0b57b3101bf443ac4f242a6908a06f2f46175a54beb\": container with ID starting with 9ab1f35c7c85fdd5f577e0b57b3101bf443ac4f242a6908a06f2f46175a54beb not found: ID does not exist" Jul 18 02:46:36.788369 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:46:36.788334 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xbd9k\" is forbidden: User \"system:node:ip-10-0-140-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-4.ec2.internal' and this object" podUID="4a816ec7-9910-4e2b-a3f1-f71f23996ce2" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xbd9k" Jul 18 02:46:36.899967 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:46:36.899933 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xbd9k\" is forbidden: User \"system:node:ip-10-0-140-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-4.ec2.internal' and this object" podUID="4a816ec7-9910-4e2b-a3f1-f71f23996ce2" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xbd9k" Jul 18 02:46:36.903934 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:36.903908 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a816ec7-9910-4e2b-a3f1-f71f23996ce2" path="/var/lib/kubelet/pods/4a816ec7-9910-4e2b-a3f1-f71f23996ce2/volumes" Jul 18 02:46:39.278287 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:39.278252 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fknfc"] Jul 18 02:46:39.282035 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:39.282013 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fknfc" Jul 18 02:46:39.313935 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:39.313906 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fknfc"] Jul 18 02:46:39.381391 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:39.381358 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj4xp\" (UniqueName: \"kubernetes.io/projected/2e4157fe-b9c3-4231-b1ec-482dbe1064ab-kube-api-access-lj4xp\") pod \"kuadrant-operator-controller-manager-86878c769f-fknfc\" (UID: \"2e4157fe-b9c3-4231-b1ec-482dbe1064ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fknfc" Jul 18 02:46:39.381555 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:39.381438 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e4157fe-b9c3-4231-b1ec-482dbe1064ab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-fknfc\" (UID: \"2e4157fe-b9c3-4231-b1ec-482dbe1064ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fknfc" Jul 18 02:46:39.482248 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:39.482215 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj4xp\" (UniqueName: \"kubernetes.io/projected/2e4157fe-b9c3-4231-b1ec-482dbe1064ab-kube-api-access-lj4xp\") pod \"kuadrant-operator-controller-manager-86878c769f-fknfc\" (UID: \"2e4157fe-b9c3-4231-b1ec-482dbe1064ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fknfc" Jul 18 02:46:39.482462 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:39.482284 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e4157fe-b9c3-4231-b1ec-482dbe1064ab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-fknfc\" (UID: \"2e4157fe-b9c3-4231-b1ec-482dbe1064ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fknfc" Jul 18 02:46:39.482777 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:39.482752 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e4157fe-b9c3-4231-b1ec-482dbe1064ab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-fknfc\" (UID: \"2e4157fe-b9c3-4231-b1ec-482dbe1064ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fknfc" Jul 18 02:46:39.497404 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:39.497377 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj4xp\" (UniqueName: \"kubernetes.io/projected/2e4157fe-b9c3-4231-b1ec-482dbe1064ab-kube-api-access-lj4xp\") pod \"kuadrant-operator-controller-manager-86878c769f-fknfc\" (UID: \"2e4157fe-b9c3-4231-b1ec-482dbe1064ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fknfc" Jul 18 02:46:39.593404 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:39.593334 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fknfc" Jul 18 02:46:40.147871 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:46:40.147843 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4157fe_b9c3_4231_b1ec_482dbe1064ab.slice/crio-a342cb873e748066992cc53a76fc76a651633a34f917cf0dbd7d1d32f6021e10 WatchSource:0}: Error finding container a342cb873e748066992cc53a76fc76a651633a34f917cf0dbd7d1d32f6021e10: Status 404 returned error can't find the container with id a342cb873e748066992cc53a76fc76a651633a34f917cf0dbd7d1d32f6021e10 Jul 18 02:46:40.151304 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:40.151274 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fknfc"] Jul 18 02:46:40.798623 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:40.798580 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-65jzn" event={"ID":"198c17a7-93e7-4928-8a23-751025f59e4d","Type":"ContainerStarted","Data":"826931cd40a14ec0fe4a1014f8229aeb091641484a43825fe8ae828bfc90a95a"} Jul 18 02:46:40.799067 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:40.798730 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-65jzn" Jul 18 02:46:40.800100 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:40.800076 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fknfc" event={"ID":"2e4157fe-b9c3-4231-b1ec-482dbe1064ab","Type":"ContainerStarted","Data":"73ca2fff44863b306899913da208578838945caa2a7639e95b4d983d233172b5"} Jul 18 02:46:40.800240 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:40.800122 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fknfc" Jul 18 02:46:40.800240 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:40.800143 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fknfc" event={"ID":"2e4157fe-b9c3-4231-b1ec-482dbe1064ab","Type":"ContainerStarted","Data":"a342cb873e748066992cc53a76fc76a651633a34f917cf0dbd7d1d32f6021e10"} Jul 18 02:46:40.836160 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:40.836109 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-65jzn" podStartSLOduration=2.172963569 podStartE2EDuration="5.836092536s" totalImagesPullingTime="3.663128967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:46:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:46:40.833695386 +0000 UTC m=+424.437343074" watchObservedRunningTime="2026-07-18 02:46:40.836092536 +0000 UTC m=+424.439740225" Jul 18 02:46:40.864336 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:40.864294 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fknfc" podStartSLOduration=1.864280268 podStartE2EDuration="1.864280268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:46:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:46:40.862714054 +0000 UTC m=+424.466361738" watchObservedRunningTime="2026-07-18 02:46:40.864280268 +0000 UTC m=+424.467927953" Jul 18 02:46:51.806737 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:51.806704 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-65jzn" Jul 18 02:46:51.807455 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:51.806774 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fknfc" Jul 18 02:46:51.888241 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:51.888213 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-65jzn"] Jul 18 02:46:51.888384 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:51.888364 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-65jzn" podUID="198c17a7-93e7-4928-8a23-751025f59e4d" containerName="manager" containerID="cri-o://826931cd40a14ec0fe4a1014f8229aeb091641484a43825fe8ae828bfc90a95a" gracePeriod=10 Jul 18 02:46:52.130867 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:52.130845 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-65jzn" Jul 18 02:46:52.172224 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:52.172197 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pslk\" (UniqueName: \"kubernetes.io/projected/198c17a7-93e7-4928-8a23-751025f59e4d-kube-api-access-9pslk\") pod \"198c17a7-93e7-4928-8a23-751025f59e4d\" (UID: \"198c17a7-93e7-4928-8a23-751025f59e4d\") " Jul 18 02:46:52.172425 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:52.172266 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/198c17a7-93e7-4928-8a23-751025f59e4d-extensions-socket-volume\") pod \"198c17a7-93e7-4928-8a23-751025f59e4d\" (UID: \"198c17a7-93e7-4928-8a23-751025f59e4d\") " Jul 18 02:46:52.172908 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:52.172887 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198c17a7-93e7-4928-8a23-751025f59e4d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "198c17a7-93e7-4928-8a23-751025f59e4d" (UID: "198c17a7-93e7-4928-8a23-751025f59e4d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 02:46:52.174289 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:52.174270 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/198c17a7-93e7-4928-8a23-751025f59e4d-kube-api-access-9pslk" (OuterVolumeSpecName: "kube-api-access-9pslk") pod "198c17a7-93e7-4928-8a23-751025f59e4d" (UID: "198c17a7-93e7-4928-8a23-751025f59e4d"). InnerVolumeSpecName "kube-api-access-9pslk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 02:46:52.273760 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:52.273737 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pslk\" (UniqueName: \"kubernetes.io/projected/198c17a7-93e7-4928-8a23-751025f59e4d-kube-api-access-9pslk\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 02:46:52.273760 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:52.273759 2585 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/198c17a7-93e7-4928-8a23-751025f59e4d-extensions-socket-volume\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 02:46:52.858992 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:52.858961 2585 generic.go:356] "Generic (PLEG): container finished" podID="198c17a7-93e7-4928-8a23-751025f59e4d" containerID="826931cd40a14ec0fe4a1014f8229aeb091641484a43825fe8ae828bfc90a95a" exitCode=0 Jul 18 02:46:52.859438 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:52.859009 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-65jzn" event={"ID":"198c17a7-93e7-4928-8a23-751025f59e4d","Type":"ContainerDied","Data":"826931cd40a14ec0fe4a1014f8229aeb091641484a43825fe8ae828bfc90a95a"} Jul 18 02:46:52.859438 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:52.859046 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-65jzn" event={"ID":"198c17a7-93e7-4928-8a23-751025f59e4d","Type":"ContainerDied","Data":"d0aa56023b308b0d44667494c15f1f23f3de69640b318c71235546892158f01c"} Jul 18 02:46:52.859438 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:52.859066 2585 scope.go:117] "RemoveContainer" containerID="826931cd40a14ec0fe4a1014f8229aeb091641484a43825fe8ae828bfc90a95a" Jul 18 02:46:52.859438 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:52.859010 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-65jzn" Jul 18 02:46:52.867767 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:52.867746 2585 scope.go:117] "RemoveContainer" containerID="826931cd40a14ec0fe4a1014f8229aeb091641484a43825fe8ae828bfc90a95a" Jul 18 02:46:52.868016 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:46:52.867999 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"826931cd40a14ec0fe4a1014f8229aeb091641484a43825fe8ae828bfc90a95a\": container with ID starting with 826931cd40a14ec0fe4a1014f8229aeb091641484a43825fe8ae828bfc90a95a not found: ID does not exist" containerID="826931cd40a14ec0fe4a1014f8229aeb091641484a43825fe8ae828bfc90a95a" Jul 18 02:46:52.868068 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:52.868024 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"826931cd40a14ec0fe4a1014f8229aeb091641484a43825fe8ae828bfc90a95a"} err="failed to get container status \"826931cd40a14ec0fe4a1014f8229aeb091641484a43825fe8ae828bfc90a95a\": rpc error: code = NotFound desc = could not find container \"826931cd40a14ec0fe4a1014f8229aeb091641484a43825fe8ae828bfc90a95a\": container with ID starting with 826931cd40a14ec0fe4a1014f8229aeb091641484a43825fe8ae828bfc90a95a not found: ID does not exist" Jul 18 02:46:52.907745 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:52.907716 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-65jzn"] Jul 18 02:46:52.921079 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:52.921047 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-65jzn"] Jul 18 02:46:54.899249 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:46:54.899218 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="198c17a7-93e7-4928-8a23-751025f59e4d" path="/var/lib/kubelet/pods/198c17a7-93e7-4928-8a23-751025f59e4d/volumes" Jul 18 02:47:10.957933 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:10.957856 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zqmbz"] Jul 18 02:47:10.958337 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:10.958245 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="198c17a7-93e7-4928-8a23-751025f59e4d" containerName="manager" Jul 18 02:47:10.958337 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:10.958259 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="198c17a7-93e7-4928-8a23-751025f59e4d" containerName="manager" Jul 18 02:47:10.958337 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:10.958330 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="198c17a7-93e7-4928-8a23-751025f59e4d" containerName="manager" Jul 18 02:47:10.962777 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:10.962755 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-zqmbz" Jul 18 02:47:10.965164 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:10.965147 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 18 02:47:10.965293 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:10.965220 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wr5wb\"" Jul 18 02:47:10.968732 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:10.968713 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zqmbz"] Jul 18 02:47:11.017460 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:11.017434 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/53081cd6-9112-4d6a-bc16-69d43eb2054f-config-file\") pod \"limitador-limitador-64c94fd64f-zqmbz\" (UID: \"53081cd6-9112-4d6a-bc16-69d43eb2054f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-zqmbz" Jul 18 02:47:11.017653 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:11.017630 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htgrk\" (UniqueName: \"kubernetes.io/projected/53081cd6-9112-4d6a-bc16-69d43eb2054f-kube-api-access-htgrk\") pod \"limitador-limitador-64c94fd64f-zqmbz\" (UID: \"53081cd6-9112-4d6a-bc16-69d43eb2054f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-zqmbz" Jul 18 02:47:11.049005 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:11.048979 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zqmbz"] Jul 18 02:47:11.118637 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:11.118612 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/53081cd6-9112-4d6a-bc16-69d43eb2054f-config-file\") pod \"limitador-limitador-64c94fd64f-zqmbz\" (UID: \"53081cd6-9112-4d6a-bc16-69d43eb2054f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-zqmbz" Jul 18 02:47:11.118743 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:11.118656 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htgrk\" (UniqueName: \"kubernetes.io/projected/53081cd6-9112-4d6a-bc16-69d43eb2054f-kube-api-access-htgrk\") pod \"limitador-limitador-64c94fd64f-zqmbz\" (UID: \"53081cd6-9112-4d6a-bc16-69d43eb2054f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-zqmbz" Jul 18 02:47:11.119411 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:11.119390 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/53081cd6-9112-4d6a-bc16-69d43eb2054f-config-file\") pod \"limitador-limitador-64c94fd64f-zqmbz\" (UID: \"53081cd6-9112-4d6a-bc16-69d43eb2054f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-zqmbz" Jul 18 02:47:11.127736 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:11.127714 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htgrk\" (UniqueName: \"kubernetes.io/projected/53081cd6-9112-4d6a-bc16-69d43eb2054f-kube-api-access-htgrk\") pod \"limitador-limitador-64c94fd64f-zqmbz\" (UID: \"53081cd6-9112-4d6a-bc16-69d43eb2054f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-zqmbz" Jul 18 02:47:11.274244 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:11.274211 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-zqmbz" Jul 18 02:47:11.400629 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:11.400606 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zqmbz"] Jul 18 02:47:11.402691 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:47:11.402663 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53081cd6_9112_4d6a_bc16_69d43eb2054f.slice/crio-34d3979c60bf95d2532480398c0443ed04274a0bf952d3be24a551097e09138a WatchSource:0}: Error finding container 34d3979c60bf95d2532480398c0443ed04274a0bf952d3be24a551097e09138a: Status 404 returned error can't find the container with id 34d3979c60bf95d2532480398c0443ed04274a0bf952d3be24a551097e09138a Jul 18 02:47:11.940674 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:11.940636 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-zqmbz" event={"ID":"53081cd6-9112-4d6a-bc16-69d43eb2054f","Type":"ContainerStarted","Data":"34d3979c60bf95d2532480398c0443ed04274a0bf952d3be24a551097e09138a"} Jul 18 02:47:14.953769 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:14.953734 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-zqmbz" event={"ID":"53081cd6-9112-4d6a-bc16-69d43eb2054f","Type":"ContainerStarted","Data":"291c7e5c32c85b6b5a564a9a27d66bf6126960c8ba90ddfcfe546437f861de10"} Jul 18 02:47:14.954204 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:14.953854 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-zqmbz" Jul 18 02:47:14.974512 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:14.974471 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-zqmbz" podStartSLOduration=2.041344631 podStartE2EDuration="4.974459283s" totalImagesPullingTime="2.933114652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:47:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:47:14.973656448 +0000 UTC m=+458.577304134" watchObservedRunningTime="2026-07-18 02:47:14.974459283 +0000 UTC m=+458.578106969" Jul 18 02:47:25.959604 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:25.959571 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-zqmbz" Jul 18 02:47:27.812443 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:27.812411 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zqmbz"] Jul 18 02:47:27.812885 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:27.812622 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-zqmbz" podUID="53081cd6-9112-4d6a-bc16-69d43eb2054f" containerName="limitador" containerID="cri-o://291c7e5c32c85b6b5a564a9a27d66bf6126960c8ba90ddfcfe546437f861de10" gracePeriod=30 Jul 18 02:47:28.359816 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:28.359794 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-zqmbz" Jul 18 02:47:28.457569 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:28.457511 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/53081cd6-9112-4d6a-bc16-69d43eb2054f-config-file\") pod \"53081cd6-9112-4d6a-bc16-69d43eb2054f\" (UID: \"53081cd6-9112-4d6a-bc16-69d43eb2054f\") " Jul 18 02:47:28.457696 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:28.457580 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htgrk\" (UniqueName: \"kubernetes.io/projected/53081cd6-9112-4d6a-bc16-69d43eb2054f-kube-api-access-htgrk\") pod \"53081cd6-9112-4d6a-bc16-69d43eb2054f\" (UID: \"53081cd6-9112-4d6a-bc16-69d43eb2054f\") " Jul 18 02:47:28.457900 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:28.457875 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53081cd6-9112-4d6a-bc16-69d43eb2054f-config-file" (OuterVolumeSpecName: "config-file") pod "53081cd6-9112-4d6a-bc16-69d43eb2054f" (UID: "53081cd6-9112-4d6a-bc16-69d43eb2054f"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 02:47:28.459459 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:28.459434 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53081cd6-9112-4d6a-bc16-69d43eb2054f-kube-api-access-htgrk" (OuterVolumeSpecName: "kube-api-access-htgrk") pod "53081cd6-9112-4d6a-bc16-69d43eb2054f" (UID: "53081cd6-9112-4d6a-bc16-69d43eb2054f"). InnerVolumeSpecName "kube-api-access-htgrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 02:47:28.558832 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:28.558809 2585 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/53081cd6-9112-4d6a-bc16-69d43eb2054f-config-file\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 02:47:28.558832 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:28.558833 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htgrk\" (UniqueName: \"kubernetes.io/projected/53081cd6-9112-4d6a-bc16-69d43eb2054f-kube-api-access-htgrk\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 02:47:29.019377 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:29.019345 2585 generic.go:356] "Generic (PLEG): container finished" podID="53081cd6-9112-4d6a-bc16-69d43eb2054f" containerID="291c7e5c32c85b6b5a564a9a27d66bf6126960c8ba90ddfcfe546437f861de10" exitCode=0 Jul 18 02:47:29.019749 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:29.019386 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-zqmbz" Jul 18 02:47:29.019749 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:29.019403 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-zqmbz" event={"ID":"53081cd6-9112-4d6a-bc16-69d43eb2054f","Type":"ContainerDied","Data":"291c7e5c32c85b6b5a564a9a27d66bf6126960c8ba90ddfcfe546437f861de10"} Jul 18 02:47:29.019749 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:29.019445 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-zqmbz" event={"ID":"53081cd6-9112-4d6a-bc16-69d43eb2054f","Type":"ContainerDied","Data":"34d3979c60bf95d2532480398c0443ed04274a0bf952d3be24a551097e09138a"} Jul 18 02:47:29.019749 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:29.019472 2585 scope.go:117] "RemoveContainer" containerID="291c7e5c32c85b6b5a564a9a27d66bf6126960c8ba90ddfcfe546437f861de10" Jul 18 02:47:29.027923 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:29.027900 2585 scope.go:117] "RemoveContainer" containerID="291c7e5c32c85b6b5a564a9a27d66bf6126960c8ba90ddfcfe546437f861de10" Jul 18 02:47:29.028159 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:47:29.028143 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"291c7e5c32c85b6b5a564a9a27d66bf6126960c8ba90ddfcfe546437f861de10\": container with ID starting with 291c7e5c32c85b6b5a564a9a27d66bf6126960c8ba90ddfcfe546437f861de10 not found: ID does not exist" containerID="291c7e5c32c85b6b5a564a9a27d66bf6126960c8ba90ddfcfe546437f861de10" Jul 18 02:47:29.028243 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:29.028167 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"291c7e5c32c85b6b5a564a9a27d66bf6126960c8ba90ddfcfe546437f861de10"} err="failed to get container status \"291c7e5c32c85b6b5a564a9a27d66bf6126960c8ba90ddfcfe546437f861de10\": rpc error: code = NotFound desc = could not find container \"291c7e5c32c85b6b5a564a9a27d66bf6126960c8ba90ddfcfe546437f861de10\": container with ID starting with 291c7e5c32c85b6b5a564a9a27d66bf6126960c8ba90ddfcfe546437f861de10 not found: ID does not exist" Jul 18 02:47:29.038118 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:29.038096 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zqmbz"] Jul 18 02:47:29.042226 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:29.042204 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zqmbz"] Jul 18 02:47:30.899675 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:30.899647 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53081cd6-9112-4d6a-bc16-69d43eb2054f" path="/var/lib/kubelet/pods/53081cd6-9112-4d6a-bc16-69d43eb2054f/volumes" Jul 18 02:47:32.647244 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:32.647205 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-2txdr"] Jul 18 02:47:32.647626 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:32.647587 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53081cd6-9112-4d6a-bc16-69d43eb2054f" containerName="limitador" Jul 18 02:47:32.647626 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:32.647602 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="53081cd6-9112-4d6a-bc16-69d43eb2054f" containerName="limitador" Jul 18 02:47:32.647696 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:32.647683 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="53081cd6-9112-4d6a-bc16-69d43eb2054f" containerName="limitador" Jul 18 02:47:32.650837 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:32.650817 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2txdr" Jul 18 02:47:32.654194 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:32.654156 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 18 02:47:32.654338 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:32.654230 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 18 02:47:32.654451 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:32.654436 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-4lwrj\"" Jul 18 02:47:32.655321 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:32.655285 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 18 02:47:32.664565 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:32.664541 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-2txdr"] Jul 18 02:47:32.791342 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:32.791315 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmszr\" (UniqueName: \"kubernetes.io/projected/891405f9-75e1-41c6-8874-04c69294207a-kube-api-access-cmszr\") pod \"postgres-69fb4594cf-2txdr\" (UID: \"891405f9-75e1-41c6-8874-04c69294207a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2txdr" Jul 18 02:47:32.791496 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:32.791374 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/891405f9-75e1-41c6-8874-04c69294207a-data\") pod \"postgres-69fb4594cf-2txdr\" (UID: \"891405f9-75e1-41c6-8874-04c69294207a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2txdr" Jul 18 02:47:32.892394 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:32.892360 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/891405f9-75e1-41c6-8874-04c69294207a-data\") pod \"postgres-69fb4594cf-2txdr\" (UID: \"891405f9-75e1-41c6-8874-04c69294207a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2txdr" Jul 18 02:47:32.892535 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:32.892420 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmszr\" (UniqueName: \"kubernetes.io/projected/891405f9-75e1-41c6-8874-04c69294207a-kube-api-access-cmszr\") pod \"postgres-69fb4594cf-2txdr\" (UID: \"891405f9-75e1-41c6-8874-04c69294207a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2txdr" Jul 18 02:47:32.892724 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:32.892706 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/891405f9-75e1-41c6-8874-04c69294207a-data\") pod \"postgres-69fb4594cf-2txdr\" (UID: \"891405f9-75e1-41c6-8874-04c69294207a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2txdr" Jul 18 02:47:32.900371 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:32.900309 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmszr\" (UniqueName: \"kubernetes.io/projected/891405f9-75e1-41c6-8874-04c69294207a-kube-api-access-cmszr\") pod \"postgres-69fb4594cf-2txdr\" (UID: \"891405f9-75e1-41c6-8874-04c69294207a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2txdr" Jul 18 02:47:32.961749 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:32.961724 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2txdr" Jul 18 02:47:33.087938 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:33.087911 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-2txdr"] Jul 18 02:47:33.090296 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:47:33.090268 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod891405f9_75e1_41c6_8874_04c69294207a.slice/crio-c99fc86f787e06a30491fdd477b7506587328649a23de03008c0f14a23464f8b WatchSource:0}: Error finding container c99fc86f787e06a30491fdd477b7506587328649a23de03008c0f14a23464f8b: Status 404 returned error can't find the container with id c99fc86f787e06a30491fdd477b7506587328649a23de03008c0f14a23464f8b Jul 18 02:47:34.047125 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:34.047040 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2txdr" event={"ID":"891405f9-75e1-41c6-8874-04c69294207a","Type":"ContainerStarted","Data":"c99fc86f787e06a30491fdd477b7506587328649a23de03008c0f14a23464f8b"} Jul 18 02:47:37.846840 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:37.846818 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 18 02:47:38.067886 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:38.067804 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2txdr" event={"ID":"891405f9-75e1-41c6-8874-04c69294207a","Type":"ContainerStarted","Data":"64d53b57016d72c8c4648d72bc617ae9bf28fd01ac470615ad5c0d438192581d"} Jul 18 02:47:38.068043 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:38.067900 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2txdr" Jul 18 02:47:38.094796 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:38.094748 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2txdr" podStartSLOduration=1.342447436 podStartE2EDuration="6.094734565s" totalImagesPullingTime="4.752287129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:47:38.092744242 +0000 UTC m=+481.696391929" watchObservedRunningTime="2026-07-18 02:47:38.094734565 +0000 UTC m=+481.698382251" Jul 18 02:47:44.102327 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:44.102298 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2txdr" Jul 18 02:47:51.814000 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:51.813966 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d6d58f489-w2gfk"] Jul 18 02:47:51.841454 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:51.841420 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d6d58f489-w2gfk"] Jul 18 02:47:51.841620 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:51.841511 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d6d58f489-w2gfk" Jul 18 02:47:51.845138 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:51.845117 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kt5jv\"" Jul 18 02:47:51.845337 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:51.845318 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 18 02:47:51.845395 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:51.845375 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 18 02:47:51.940624 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:51.940597 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d46tk\" (UniqueName: \"kubernetes.io/projected/92623ee5-bdf3-4973-944e-ae6662d327ac-kube-api-access-d46tk\") pod \"maas-controller-d6d58f489-w2gfk\" (UID: \"92623ee5-bdf3-4973-944e-ae6662d327ac\") " pod="opendatahub/maas-controller-d6d58f489-w2gfk" Jul 18 02:47:51.940750 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:51.940653 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92623ee5-bdf3-4973-944e-ae6662d327ac-webhook-cert\") pod \"maas-controller-d6d58f489-w2gfk\" (UID: \"92623ee5-bdf3-4973-944e-ae6662d327ac\") " pod="opendatahub/maas-controller-d6d58f489-w2gfk" Jul 18 02:47:52.041257 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:52.041230 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92623ee5-bdf3-4973-944e-ae6662d327ac-webhook-cert\") pod \"maas-controller-d6d58f489-w2gfk\" (UID: \"92623ee5-bdf3-4973-944e-ae6662d327ac\") " pod="opendatahub/maas-controller-d6d58f489-w2gfk" Jul 18 02:47:52.041365 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:52.041282 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d46tk\" (UniqueName: \"kubernetes.io/projected/92623ee5-bdf3-4973-944e-ae6662d327ac-kube-api-access-d46tk\") pod \"maas-controller-d6d58f489-w2gfk\" (UID: \"92623ee5-bdf3-4973-944e-ae6662d327ac\") " pod="opendatahub/maas-controller-d6d58f489-w2gfk" Jul 18 02:47:52.043791 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:52.043771 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92623ee5-bdf3-4973-944e-ae6662d327ac-webhook-cert\") pod \"maas-controller-d6d58f489-w2gfk\" (UID: \"92623ee5-bdf3-4973-944e-ae6662d327ac\") " pod="opendatahub/maas-controller-d6d58f489-w2gfk" Jul 18 02:47:52.050459 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:52.050437 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d46tk\" (UniqueName: \"kubernetes.io/projected/92623ee5-bdf3-4973-944e-ae6662d327ac-kube-api-access-d46tk\") pod \"maas-controller-d6d58f489-w2gfk\" (UID: \"92623ee5-bdf3-4973-944e-ae6662d327ac\") " pod="opendatahub/maas-controller-d6d58f489-w2gfk" Jul 18 02:47:52.151333 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:52.151281 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d6d58f489-w2gfk" Jul 18 02:47:52.275690 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:52.275662 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d6d58f489-w2gfk"] Jul 18 02:47:52.277627 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:47:52.277603 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92623ee5_bdf3_4973_944e_ae6662d327ac.slice/crio-abd74c284c5d476597e4b4e1e6dcb4b4bdc519a3ce382dccfadd4365149ec75d WatchSource:0}: Error finding container abd74c284c5d476597e4b4e1e6dcb4b4bdc519a3ce382dccfadd4365149ec75d: Status 404 returned error can't find the container with id abd74c284c5d476597e4b4e1e6dcb4b4bdc519a3ce382dccfadd4365149ec75d Jul 18 02:47:53.139737 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:53.139695 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d6d58f489-w2gfk" event={"ID":"92623ee5-bdf3-4973-944e-ae6662d327ac","Type":"ContainerStarted","Data":"abd74c284c5d476597e4b4e1e6dcb4b4bdc519a3ce382dccfadd4365149ec75d"} Jul 18 02:47:55.155205 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:55.155146 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d6d58f489-w2gfk" event={"ID":"92623ee5-bdf3-4973-944e-ae6662d327ac","Type":"ContainerStarted","Data":"20b23bb70d022bbe394a5c20fee55ac7c236769827425226fec68ba960b4cee4"} Jul 18 02:47:55.155670 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:55.155248 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d6d58f489-w2gfk" Jul 18 02:47:55.175831 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:47:55.175786 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d6d58f489-w2gfk" podStartSLOduration=1.92266696 podStartE2EDuration="4.175771409s" totalImagesPullingTime="2.253104449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:47:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:47:55.175454553 +0000 UTC m=+498.779102239" watchObservedRunningTime="2026-07-18 02:47:55.175771409 +0000 UTC m=+498.779419094" Jul 18 02:48:06.161748 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:06.161716 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d6d58f489-w2gfk" Jul 18 02:48:27.093606 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:27.093573 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5cdfd7f48c-smpvr"] Jul 18 02:48:27.097011 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:27.096995 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cdfd7f48c-smpvr" Jul 18 02:48:27.101684 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:27.101666 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 18 02:48:27.101759 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:27.101750 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8dwff\"" Jul 18 02:48:27.108302 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:27.108282 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cdfd7f48c-smpvr"] Jul 18 02:48:27.207809 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:27.207784 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnz4s\" (UniqueName: \"kubernetes.io/projected/ca4f177c-0e56-47a2-913c-d0668358122c-kube-api-access-qnz4s\") pod \"maas-api-5cdfd7f48c-smpvr\" (UID: \"ca4f177c-0e56-47a2-913c-d0668358122c\") " pod="odh-ai-gateway-infra/maas-api-5cdfd7f48c-smpvr" Jul 18 02:48:27.207922 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:27.207825 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca4f177c-0e56-47a2-913c-d0668358122c-maas-api-tls\") pod \"maas-api-5cdfd7f48c-smpvr\" (UID: \"ca4f177c-0e56-47a2-913c-d0668358122c\") " pod="odh-ai-gateway-infra/maas-api-5cdfd7f48c-smpvr" Jul 18 02:48:27.207922 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:27.207892 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca4f177c-0e56-47a2-913c-d0668358122c-openshift-service-ca\") pod \"maas-api-5cdfd7f48c-smpvr\" (UID: \"ca4f177c-0e56-47a2-913c-d0668358122c\") " pod="odh-ai-gateway-infra/maas-api-5cdfd7f48c-smpvr" Jul 18 02:48:27.308985 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:27.308954 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca4f177c-0e56-47a2-913c-d0668358122c-openshift-service-ca\") pod \"maas-api-5cdfd7f48c-smpvr\" (UID: \"ca4f177c-0e56-47a2-913c-d0668358122c\") " pod="odh-ai-gateway-infra/maas-api-5cdfd7f48c-smpvr" Jul 18 02:48:27.309127 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:27.309003 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnz4s\" (UniqueName: \"kubernetes.io/projected/ca4f177c-0e56-47a2-913c-d0668358122c-kube-api-access-qnz4s\") pod \"maas-api-5cdfd7f48c-smpvr\" (UID: \"ca4f177c-0e56-47a2-913c-d0668358122c\") " pod="odh-ai-gateway-infra/maas-api-5cdfd7f48c-smpvr" Jul 18 02:48:27.309127 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:27.309034 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca4f177c-0e56-47a2-913c-d0668358122c-maas-api-tls\") pod \"maas-api-5cdfd7f48c-smpvr\" (UID: \"ca4f177c-0e56-47a2-913c-d0668358122c\") " pod="odh-ai-gateway-infra/maas-api-5cdfd7f48c-smpvr" Jul 18 02:48:27.309251 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:48:27.309128 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 18 02:48:27.309251 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:48:27.309207 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca4f177c-0e56-47a2-913c-d0668358122c-maas-api-tls podName:ca4f177c-0e56-47a2-913c-d0668358122c nodeName:}" failed. No retries permitted until 2026-07-18 02:48:27.809168119 +0000 UTC m=+531.412815786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ca4f177c-0e56-47a2-913c-d0668358122c-maas-api-tls") pod "maas-api-5cdfd7f48c-smpvr" (UID: "ca4f177c-0e56-47a2-913c-d0668358122c") : secret "maas-api-serving-cert" not found Jul 18 02:48:27.309674 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:27.309654 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca4f177c-0e56-47a2-913c-d0668358122c-openshift-service-ca\") pod \"maas-api-5cdfd7f48c-smpvr\" (UID: \"ca4f177c-0e56-47a2-913c-d0668358122c\") " pod="odh-ai-gateway-infra/maas-api-5cdfd7f48c-smpvr" Jul 18 02:48:27.318758 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:27.318737 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnz4s\" (UniqueName: \"kubernetes.io/projected/ca4f177c-0e56-47a2-913c-d0668358122c-kube-api-access-qnz4s\") pod \"maas-api-5cdfd7f48c-smpvr\" (UID: \"ca4f177c-0e56-47a2-913c-d0668358122c\") " pod="odh-ai-gateway-infra/maas-api-5cdfd7f48c-smpvr" Jul 18 02:48:27.814539 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:27.814504 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca4f177c-0e56-47a2-913c-d0668358122c-maas-api-tls\") pod \"maas-api-5cdfd7f48c-smpvr\" (UID: \"ca4f177c-0e56-47a2-913c-d0668358122c\") " pod="odh-ai-gateway-infra/maas-api-5cdfd7f48c-smpvr" Jul 18 02:48:27.816760 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:27.816736 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca4f177c-0e56-47a2-913c-d0668358122c-maas-api-tls\") pod \"maas-api-5cdfd7f48c-smpvr\" (UID: \"ca4f177c-0e56-47a2-913c-d0668358122c\") " pod="odh-ai-gateway-infra/maas-api-5cdfd7f48c-smpvr" Jul 18 02:48:28.007484 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:28.007452 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cdfd7f48c-smpvr" Jul 18 02:48:28.134701 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:28.134674 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cdfd7f48c-smpvr"] Jul 18 02:48:28.136622 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:48:28.136590 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca4f177c_0e56_47a2_913c_d0668358122c.slice/crio-212a42a54fc4617d10b3584b4ba6c261771552872647bfcfdf15d6e7dae1642d WatchSource:0}: Error finding container 212a42a54fc4617d10b3584b4ba6c261771552872647bfcfdf15d6e7dae1642d: Status 404 returned error can't find the container with id 212a42a54fc4617d10b3584b4ba6c261771552872647bfcfdf15d6e7dae1642d Jul 18 02:48:28.297849 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:28.297813 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cdfd7f48c-smpvr" event={"ID":"ca4f177c-0e56-47a2-913c-d0668358122c","Type":"ContainerStarted","Data":"212a42a54fc4617d10b3584b4ba6c261771552872647bfcfdf15d6e7dae1642d"} Jul 18 02:48:30.309020 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:30.308992 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cdfd7f48c-smpvr" event={"ID":"ca4f177c-0e56-47a2-913c-d0668358122c","Type":"ContainerStarted","Data":"3455474855df4a7f12bdc780b379b2d32bbc2813687540e23f0ec96cc4123b12"} Jul 18 02:48:30.309386 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:30.309097 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5cdfd7f48c-smpvr" Jul 18 02:48:30.326842 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:30.326797 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5cdfd7f48c-smpvr" podStartSLOduration=1.975444382 podStartE2EDuration="3.326785394s" totalImagesPullingTime="1.351341012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:48:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:48:30.325201511 +0000 UTC m=+533.928849194" watchObservedRunningTime="2026-07-18 02:48:30.326785394 +0000 UTC m=+533.930433079" Jul 18 02:48:36.319950 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:36.319918 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5cdfd7f48c-smpvr" Jul 18 02:48:49.102002 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:49.101969 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2"] Jul 18 02:48:49.105923 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:49.105903 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2" Jul 18 02:48:49.108404 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:49.108385 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 18 02:48:49.108404 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:49.108394 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 18 02:48:49.108565 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:49.108428 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 18 02:48:49.109474 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:49.109455 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-j4nw7\"" Jul 18 02:48:49.115150 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:49.115129 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2"] Jul 18 02:48:49.272534 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:49.272501 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14212281-7994-4954-8913-986f89a0bdaf-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2\" (UID: \"14212281-7994-4954-8913-986f89a0bdaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2" Jul 18 02:48:49.272534 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:49.272536 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14212281-7994-4954-8913-986f89a0bdaf-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2\" (UID: \"14212281-7994-4954-8913-986f89a0bdaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2" Jul 18 02:48:49.272723 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:49.272560 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14212281-7994-4954-8913-986f89a0bdaf-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2\" (UID: \"14212281-7994-4954-8913-986f89a0bdaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2" Jul 18 02:48:49.272723 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:49.272661 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14212281-7994-4954-8913-986f89a0bdaf-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2\" (UID: \"14212281-7994-4954-8913-986f89a0bdaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2" Jul 18 02:48:49.272723 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:49.272697 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14212281-7994-4954-8913-986f89a0bdaf-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2\" (UID: \"14212281-7994-4954-8913-986f89a0bdaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2" Jul 18 02:48:49.272823 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:49.272729 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vlsg\" (UniqueName: \"kubernetes.io/projected/14212281-7994-4954-8913-986f89a0bdaf-kube-api-access-4vlsg\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2\" (UID: \"14212281-7994-4954-8913-986f89a0bdaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2" Jul 18 02:48:49.373629 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:49.373549 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14212281-7994-4954-8913-986f89a0bdaf-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2\" (UID: \"14212281-7994-4954-8913-986f89a0bdaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2" Jul 18 02:48:49.373629 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:49.373588 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14212281-7994-4954-8913-986f89a0bdaf-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2\" (UID: \"14212281-7994-4954-8913-986f89a0bdaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2" Jul 18 02:48:49.373629 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:49.373611 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vlsg\" (UniqueName: \"kubernetes.io/projected/14212281-7994-4954-8913-986f89a0bdaf-kube-api-access-4vlsg\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2\" (UID: \"14212281-7994-4954-8913-986f89a0bdaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2" Jul 18 02:48:49.373863 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:49.373742 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14212281-7994-4954-8913-986f89a0bdaf-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2\" (UID: \"14212281-7994-4954-8913-986f89a0bdaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2" Jul 18 02:48:49.373863 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:49.373783 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14212281-7994-4954-8913-986f89a0bdaf-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2\" (UID: \"14212281-7994-4954-8913-986f89a0bdaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2" Jul 18 02:48:49.373972 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:49.373900 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14212281-7994-4954-8913-986f89a0bdaf-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2\" (UID: \"14212281-7994-4954-8913-986f89a0bdaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2" Jul 18 02:48:49.373972 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:49.373916 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14212281-7994-4954-8913-986f89a0bdaf-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2\" (UID: \"14212281-7994-4954-8913-986f89a0bdaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2" Jul 18 02:48:49.374241 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:49.374103 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14212281-7994-4954-8913-986f89a0bdaf-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2\" (UID: \"14212281-7994-4954-8913-986f89a0bdaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2" Jul 18 02:48:49.374241 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:49.374149 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14212281-7994-4954-8913-986f89a0bdaf-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2\" (UID: \"14212281-7994-4954-8913-986f89a0bdaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2" Jul 18 02:48:49.375909 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:49.375881 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14212281-7994-4954-8913-986f89a0bdaf-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2\" (UID: \"14212281-7994-4954-8913-986f89a0bdaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2" Jul 18 02:48:49.376158 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:49.376142 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14212281-7994-4954-8913-986f89a0bdaf-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2\" (UID: \"14212281-7994-4954-8913-986f89a0bdaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2" Jul 18 02:48:49.381398 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:49.381374 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vlsg\" (UniqueName: \"kubernetes.io/projected/14212281-7994-4954-8913-986f89a0bdaf-kube-api-access-4vlsg\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2\" (UID: \"14212281-7994-4954-8913-986f89a0bdaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2" Jul 18 02:48:49.416538 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:49.416514 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2" Jul 18 02:48:49.549239 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:49.549208 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2"] Jul 18 02:48:49.550618 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:48:49.550590 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14212281_7994_4954_8913_986f89a0bdaf.slice/crio-f8677ad30dea427c913c7aacacba4f2bdf88c096a62d6f954b86f65b886f4edc WatchSource:0}: Error finding container f8677ad30dea427c913c7aacacba4f2bdf88c096a62d6f954b86f65b886f4edc: Status 404 returned error can't find the container with id f8677ad30dea427c913c7aacacba4f2bdf88c096a62d6f954b86f65b886f4edc Jul 18 02:48:50.406043 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:50.406009 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2" event={"ID":"14212281-7994-4954-8913-986f89a0bdaf","Type":"ContainerStarted","Data":"f8677ad30dea427c913c7aacacba4f2bdf88c096a62d6f954b86f65b886f4edc"} Jul 18 02:48:51.411436 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:51.411402 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2" event={"ID":"14212281-7994-4954-8913-986f89a0bdaf","Type":"ContainerStarted","Data":"ad5eb81d527ad51c0377ebed37fc218d72ba6191a72a96f9a339c0194459791e"} Jul 18 02:48:51.429076 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:51.429025 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2" podStartSLOduration=1.3436721710000001 podStartE2EDuration="2.429010875s" totalImagesPullingTime="1.085338704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:48:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:48:51.427560736 +0000 UTC m=+555.031208423" watchObservedRunningTime="2026-07-18 02:48:51.429010875 +0000 UTC m=+555.032658560" Jul 18 02:48:57.071379 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:57.071343 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-szbrt"] Jul 18 02:48:57.074841 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:57.074824 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-szbrt" Jul 18 02:48:57.078971 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:57.078948 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 18 02:48:57.088162 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:57.088140 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-szbrt"] Jul 18 02:48:57.238436 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:57.238404 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25ade0b7-80a2-4f51-8c12-2f917c1d164e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-szbrt\" (UID: \"25ade0b7-80a2-4f51-8c12-2f917c1d164e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-szbrt" Jul 18 02:48:57.238436 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:57.238438 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25ade0b7-80a2-4f51-8c12-2f917c1d164e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-szbrt\" (UID: \"25ade0b7-80a2-4f51-8c12-2f917c1d164e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-szbrt" Jul 18 02:48:57.238588 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:57.238461 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25ade0b7-80a2-4f51-8c12-2f917c1d164e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-szbrt\" (UID: \"25ade0b7-80a2-4f51-8c12-2f917c1d164e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-szbrt" Jul 18 02:48:57.238588 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:57.238483 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5wfc\" (UniqueName: \"kubernetes.io/projected/25ade0b7-80a2-4f51-8c12-2f917c1d164e-kube-api-access-h5wfc\") pod \"e2e-distinct-simulated-kserve-59975d69f9-szbrt\" (UID: \"25ade0b7-80a2-4f51-8c12-2f917c1d164e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-szbrt" Jul 18 02:48:57.238588 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:57.238536 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25ade0b7-80a2-4f51-8c12-2f917c1d164e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-szbrt\" (UID: \"25ade0b7-80a2-4f51-8c12-2f917c1d164e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-szbrt" Jul 18 02:48:57.238588 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:57.238574 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25ade0b7-80a2-4f51-8c12-2f917c1d164e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-szbrt\" (UID: \"25ade0b7-80a2-4f51-8c12-2f917c1d164e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-szbrt" Jul 18 02:48:57.339696 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:57.339623 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25ade0b7-80a2-4f51-8c12-2f917c1d164e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-szbrt\" (UID: \"25ade0b7-80a2-4f51-8c12-2f917c1d164e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-szbrt" Jul 18 02:48:57.339696 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:57.339665 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25ade0b7-80a2-4f51-8c12-2f917c1d164e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-szbrt\" (UID: \"25ade0b7-80a2-4f51-8c12-2f917c1d164e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-szbrt" Jul 18 02:48:57.339882 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:57.339744 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25ade0b7-80a2-4f51-8c12-2f917c1d164e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-szbrt\" (UID: \"25ade0b7-80a2-4f51-8c12-2f917c1d164e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-szbrt" Jul 18 02:48:57.339882 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:57.339770 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25ade0b7-80a2-4f51-8c12-2f917c1d164e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-szbrt\" (UID: \"25ade0b7-80a2-4f51-8c12-2f917c1d164e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-szbrt" Jul 18 02:48:57.339882 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:57.339798 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25ade0b7-80a2-4f51-8c12-2f917c1d164e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-szbrt\" (UID: \"25ade0b7-80a2-4f51-8c12-2f917c1d164e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-szbrt" Jul 18 02:48:57.340028 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:57.339927 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5wfc\" (UniqueName: \"kubernetes.io/projected/25ade0b7-80a2-4f51-8c12-2f917c1d164e-kube-api-access-h5wfc\") pod \"e2e-distinct-simulated-kserve-59975d69f9-szbrt\" (UID: \"25ade0b7-80a2-4f51-8c12-2f917c1d164e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-szbrt" Jul 18 02:48:57.340072 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:57.340041 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25ade0b7-80a2-4f51-8c12-2f917c1d164e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-szbrt\" (UID: \"25ade0b7-80a2-4f51-8c12-2f917c1d164e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-szbrt" Jul 18 02:48:57.340113 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:57.340096 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25ade0b7-80a2-4f51-8c12-2f917c1d164e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-szbrt\" (UID: \"25ade0b7-80a2-4f51-8c12-2f917c1d164e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-szbrt" Jul 18 02:48:57.340359 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:57.340335 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25ade0b7-80a2-4f51-8c12-2f917c1d164e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-szbrt\" (UID: \"25ade0b7-80a2-4f51-8c12-2f917c1d164e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-szbrt" Jul 18 02:48:57.342091 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:57.342069 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25ade0b7-80a2-4f51-8c12-2f917c1d164e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-szbrt\" (UID: \"25ade0b7-80a2-4f51-8c12-2f917c1d164e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-szbrt" Jul 18 02:48:57.342286 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:57.342267 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25ade0b7-80a2-4f51-8c12-2f917c1d164e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-szbrt\" (UID: \"25ade0b7-80a2-4f51-8c12-2f917c1d164e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-szbrt" Jul 18 02:48:57.353305 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:57.353279 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5wfc\" (UniqueName: \"kubernetes.io/projected/25ade0b7-80a2-4f51-8c12-2f917c1d164e-kube-api-access-h5wfc\") pod \"e2e-distinct-simulated-kserve-59975d69f9-szbrt\" (UID: \"25ade0b7-80a2-4f51-8c12-2f917c1d164e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-szbrt" Jul 18 02:48:57.385604 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:57.385582 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-szbrt" Jul 18 02:48:57.515095 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:57.515061 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-szbrt"] Jul 18 02:48:57.516540 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:48:57.516512 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ade0b7_80a2_4f51_8c12_2f917c1d164e.slice/crio-5f1a916476a52b1736e121fcbb95213b27c2390f10bd1d031bfb51b89edc6688 WatchSource:0}: Error finding container 5f1a916476a52b1736e121fcbb95213b27c2390f10bd1d031bfb51b89edc6688: Status 404 returned error can't find the container with id 5f1a916476a52b1736e121fcbb95213b27c2390f10bd1d031bfb51b89edc6688 Jul 18 02:48:58.444626 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:58.444594 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-szbrt" event={"ID":"25ade0b7-80a2-4f51-8c12-2f917c1d164e","Type":"ContainerStarted","Data":"382b290c655e5548460031bc5f4aa199207aee0d405d69a8ed4c82ea7f97b6f9"} Jul 18 02:48:58.445072 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:58.444642 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-szbrt" event={"ID":"25ade0b7-80a2-4f51-8c12-2f917c1d164e","Type":"ContainerStarted","Data":"5f1a916476a52b1736e121fcbb95213b27c2390f10bd1d031bfb51b89edc6688"} Jul 18 02:48:58.462978 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:58.462936 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-szbrt" podStartSLOduration=1.311801357 podStartE2EDuration="1.462920775s" totalImagesPullingTime="151.119418ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:48:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:48:58.461360708 +0000 UTC m=+562.065008395" watchObservedRunningTime="2026-07-18 02:48:58.462920775 +0000 UTC m=+562.066568461" Jul 18 02:48:59.416812 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:59.416780 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2" Jul 18 02:48:59.416968 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:59.416844 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2" Jul 18 02:48:59.429237 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:59.429211 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2" Jul 18 02:48:59.464910 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:48:59.464891 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2" Jul 18 02:49:07.385878 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:07.385838 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-szbrt" Jul 18 02:49:07.386301 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:07.385901 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-szbrt" Jul 18 02:49:07.398158 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:07.398131 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-szbrt" Jul 18 02:49:07.495538 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:07.495516 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-szbrt" Jul 18 02:49:36.684647 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:36.684612 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zn6pd"] Jul 18 02:49:36.688455 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:36.688436 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zn6pd" Jul 18 02:49:36.690890 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:36.690871 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 18 02:49:36.696399 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:36.696377 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zn6pd"] Jul 18 02:49:36.748049 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:36.748021 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e778bb49-179d-4c76-8984-e1bdb8d1cb86-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zn6pd\" (UID: \"e778bb49-179d-4c76-8984-e1bdb8d1cb86\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zn6pd" Jul 18 02:49:36.748210 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:36.748078 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp865\" (UniqueName: \"kubernetes.io/projected/e778bb49-179d-4c76-8984-e1bdb8d1cb86-kube-api-access-sp865\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zn6pd\" (UID: \"e778bb49-179d-4c76-8984-e1bdb8d1cb86\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zn6pd" Jul 18 02:49:36.748210 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:36.748113 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e778bb49-179d-4c76-8984-e1bdb8d1cb86-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zn6pd\" (UID: \"e778bb49-179d-4c76-8984-e1bdb8d1cb86\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zn6pd" Jul 18 02:49:36.748210 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:36.748140 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e778bb49-179d-4c76-8984-e1bdb8d1cb86-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zn6pd\" (UID: \"e778bb49-179d-4c76-8984-e1bdb8d1cb86\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zn6pd" Jul 18 02:49:36.748210 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:36.748156 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e778bb49-179d-4c76-8984-e1bdb8d1cb86-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zn6pd\" (UID: \"e778bb49-179d-4c76-8984-e1bdb8d1cb86\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zn6pd" Jul 18 02:49:36.748210 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:36.748179 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e778bb49-179d-4c76-8984-e1bdb8d1cb86-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zn6pd\" (UID: \"e778bb49-179d-4c76-8984-e1bdb8d1cb86\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zn6pd" Jul 18 02:49:36.848647 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:36.848616 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp865\" (UniqueName: \"kubernetes.io/projected/e778bb49-179d-4c76-8984-e1bdb8d1cb86-kube-api-access-sp865\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zn6pd\" (UID: \"e778bb49-179d-4c76-8984-e1bdb8d1cb86\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zn6pd" Jul 18 02:49:36.848837 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:36.848668 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e778bb49-179d-4c76-8984-e1bdb8d1cb86-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zn6pd\" (UID: \"e778bb49-179d-4c76-8984-e1bdb8d1cb86\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zn6pd" Jul 18 02:49:36.848837 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:36.848705 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e778bb49-179d-4c76-8984-e1bdb8d1cb86-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zn6pd\" (UID: \"e778bb49-179d-4c76-8984-e1bdb8d1cb86\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zn6pd" Jul 18 02:49:36.848837 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:36.848728 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e778bb49-179d-4c76-8984-e1bdb8d1cb86-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zn6pd\" (UID: \"e778bb49-179d-4c76-8984-e1bdb8d1cb86\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zn6pd" Jul 18 02:49:36.848837 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:36.848771 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e778bb49-179d-4c76-8984-e1bdb8d1cb86-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zn6pd\" (UID: \"e778bb49-179d-4c76-8984-e1bdb8d1cb86\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zn6pd" Jul 18 02:49:36.849037 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:36.848840 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e778bb49-179d-4c76-8984-e1bdb8d1cb86-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zn6pd\" (UID: \"e778bb49-179d-4c76-8984-e1bdb8d1cb86\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zn6pd" Jul 18 02:49:36.849093 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:36.849059 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e778bb49-179d-4c76-8984-e1bdb8d1cb86-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zn6pd\" (UID: \"e778bb49-179d-4c76-8984-e1bdb8d1cb86\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zn6pd" Jul 18 02:49:36.849093 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:36.849075 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e778bb49-179d-4c76-8984-e1bdb8d1cb86-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zn6pd\" (UID: \"e778bb49-179d-4c76-8984-e1bdb8d1cb86\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zn6pd" Jul 18 02:49:36.849286 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:36.849266 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e778bb49-179d-4c76-8984-e1bdb8d1cb86-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zn6pd\" (UID: \"e778bb49-179d-4c76-8984-e1bdb8d1cb86\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zn6pd" Jul 18 02:49:36.850991 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:36.850967 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 18 02:49:36.851564 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:36.851543 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e778bb49-179d-4c76-8984-e1bdb8d1cb86-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zn6pd\" (UID: \"e778bb49-179d-4c76-8984-e1bdb8d1cb86\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zn6pd" Jul 18 02:49:36.852707 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:36.852685 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-krtsg_709575dc-6eae-43e1-9208-cc228795929a/console-operator/1.log" Jul 18 02:49:36.853403 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:36.853388 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-krtsg_709575dc-6eae-43e1-9208-cc228795929a/console-operator/1.log" Jul 18 02:49:36.858814 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:36.858791 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp865\" (UniqueName: \"kubernetes.io/projected/e778bb49-179d-4c76-8984-e1bdb8d1cb86-kube-api-access-sp865\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zn6pd\" (UID: \"e778bb49-179d-4c76-8984-e1bdb8d1cb86\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zn6pd" Jul 18 02:49:36.861508 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:36.861480 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e778bb49-179d-4c76-8984-e1bdb8d1cb86-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zn6pd\" (UID: \"e778bb49-179d-4c76-8984-e1bdb8d1cb86\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zn6pd" Jul 18 02:49:36.999588 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:36.999562 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zn6pd" Jul 18 02:49:37.137013 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:37.136983 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zn6pd"] Jul 18 02:49:37.139554 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:49:37.139524 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode778bb49_179d_4c76_8984_e1bdb8d1cb86.slice/crio-541f51a8dad57f389284fe14ee0e42b7d8652575431137746706ee3bf4f4e560 WatchSource:0}: Error finding container 541f51a8dad57f389284fe14ee0e42b7d8652575431137746706ee3bf4f4e560: Status 404 returned error can't find the container with id 541f51a8dad57f389284fe14ee0e42b7d8652575431137746706ee3bf4f4e560 Jul 18 02:49:37.626651 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:37.626569 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zn6pd" event={"ID":"e778bb49-179d-4c76-8984-e1bdb8d1cb86","Type":"ContainerStarted","Data":"c3330b617d1b794bd98c48aae8cafbee519bc4b0c2c282c1e2278ee1ba577610"} Jul 18 02:49:37.626651 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:37.626621 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zn6pd" event={"ID":"e778bb49-179d-4c76-8984-e1bdb8d1cb86","Type":"ContainerStarted","Data":"541f51a8dad57f389284fe14ee0e42b7d8652575431137746706ee3bf4f4e560"} Jul 18 02:49:37.645472 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:37.645410 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zn6pd" podStartSLOduration=1.493977074 podStartE2EDuration="1.64539536s" totalImagesPullingTime="151.418286ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:49:37.644721833 +0000 UTC m=+601.248369525" watchObservedRunningTime="2026-07-18 02:49:37.64539536 +0000 UTC m=+601.249043046" Jul 18 02:49:47.000221 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:47.000172 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zn6pd" Jul 18 02:49:47.002690 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:47.000247 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zn6pd" Jul 18 02:49:47.012405 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:47.012381 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zn6pd" Jul 18 02:49:47.685631 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:49:47.685607 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zn6pd" Jul 18 02:51:50.150319 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:51:50.150281 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d6d58f489-w2gfk"] Jul 18 02:51:50.150798 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:51:50.150473 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-d6d58f489-w2gfk" podUID="92623ee5-bdf3-4973-944e-ae6662d327ac" containerName="manager" containerID="cri-o://20b23bb70d022bbe394a5c20fee55ac7c236769827425226fec68ba960b4cee4" gracePeriod=30 Jul 18 02:51:50.398276 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:51:50.398252 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d6d58f489-w2gfk" Jul 18 02:51:50.521681 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:51:50.521648 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d46tk\" (UniqueName: \"kubernetes.io/projected/92623ee5-bdf3-4973-944e-ae6662d327ac-kube-api-access-d46tk\") pod \"92623ee5-bdf3-4973-944e-ae6662d327ac\" (UID: \"92623ee5-bdf3-4973-944e-ae6662d327ac\") " Jul 18 02:51:50.521852 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:51:50.521721 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92623ee5-bdf3-4973-944e-ae6662d327ac-webhook-cert\") pod \"92623ee5-bdf3-4973-944e-ae6662d327ac\" (UID: \"92623ee5-bdf3-4973-944e-ae6662d327ac\") " Jul 18 02:51:50.523917 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:51:50.523890 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92623ee5-bdf3-4973-944e-ae6662d327ac-kube-api-access-d46tk" (OuterVolumeSpecName: "kube-api-access-d46tk") pod "92623ee5-bdf3-4973-944e-ae6662d327ac" (UID: "92623ee5-bdf3-4973-944e-ae6662d327ac"). InnerVolumeSpecName "kube-api-access-d46tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 02:51:50.524024 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:51:50.523997 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92623ee5-bdf3-4973-944e-ae6662d327ac-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "92623ee5-bdf3-4973-944e-ae6662d327ac" (UID: "92623ee5-bdf3-4973-944e-ae6662d327ac"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 02:51:50.623352 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:51:50.623317 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d46tk\" (UniqueName: \"kubernetes.io/projected/92623ee5-bdf3-4973-944e-ae6662d327ac-kube-api-access-d46tk\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 02:51:50.623352 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:51:50.623348 2585 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92623ee5-bdf3-4973-944e-ae6662d327ac-webhook-cert\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 02:51:51.256597 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:51:51.256558 2585 generic.go:356] "Generic (PLEG): container finished" podID="92623ee5-bdf3-4973-944e-ae6662d327ac" containerID="20b23bb70d022bbe394a5c20fee55ac7c236769827425226fec68ba960b4cee4" exitCode=0 Jul 18 02:51:51.256997 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:51:51.256639 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d6d58f489-w2gfk" Jul 18 02:51:51.256997 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:51:51.256648 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d6d58f489-w2gfk" event={"ID":"92623ee5-bdf3-4973-944e-ae6662d327ac","Type":"ContainerDied","Data":"20b23bb70d022bbe394a5c20fee55ac7c236769827425226fec68ba960b4cee4"} Jul 18 02:51:51.256997 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:51:51.256701 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d6d58f489-w2gfk" event={"ID":"92623ee5-bdf3-4973-944e-ae6662d327ac","Type":"ContainerDied","Data":"abd74c284c5d476597e4b4e1e6dcb4b4bdc519a3ce382dccfadd4365149ec75d"} Jul 18 02:51:51.256997 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:51:51.256721 2585 scope.go:117] "RemoveContainer" containerID="20b23bb70d022bbe394a5c20fee55ac7c236769827425226fec68ba960b4cee4" Jul 18 02:51:51.265507 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:51:51.265474 2585 scope.go:117] "RemoveContainer" containerID="20b23bb70d022bbe394a5c20fee55ac7c236769827425226fec68ba960b4cee4" Jul 18 02:51:51.265745 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:51:51.265723 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20b23bb70d022bbe394a5c20fee55ac7c236769827425226fec68ba960b4cee4\": container with ID starting with 20b23bb70d022bbe394a5c20fee55ac7c236769827425226fec68ba960b4cee4 not found: ID does not exist" containerID="20b23bb70d022bbe394a5c20fee55ac7c236769827425226fec68ba960b4cee4" Jul 18 02:51:51.265814 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:51:51.265758 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20b23bb70d022bbe394a5c20fee55ac7c236769827425226fec68ba960b4cee4"} err="failed to get container status \"20b23bb70d022bbe394a5c20fee55ac7c236769827425226fec68ba960b4cee4\": rpc error: code = NotFound desc = could not find container \"20b23bb70d022bbe394a5c20fee55ac7c236769827425226fec68ba960b4cee4\": container with ID starting with 20b23bb70d022bbe394a5c20fee55ac7c236769827425226fec68ba960b4cee4 not found: ID does not exist" Jul 18 02:51:51.273651 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:51:51.273617 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d6d58f489-w2gfk"] Jul 18 02:51:51.277513 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:51:51.277490 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-d6d58f489-w2gfk"] Jul 18 02:51:52.900343 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:51:52.900310 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92623ee5-bdf3-4973-944e-ae6662d327ac" path="/var/lib/kubelet/pods/92623ee5-bdf3-4973-944e-ae6662d327ac/volumes" Jul 18 02:54:36.884889 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:54:36.884811 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-krtsg_709575dc-6eae-43e1-9208-cc228795929a/console-operator/1.log" Jul 18 02:54:36.894385 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:54:36.894358 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-krtsg_709575dc-6eae-43e1-9208-cc228795929a/console-operator/1.log" Jul 18 02:57:55.799136 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:57:55.799099 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fknfc"] Jul 18 02:57:55.799658 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:57:55.799336 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fknfc" podUID="2e4157fe-b9c3-4231-b1ec-482dbe1064ab" containerName="manager" containerID="cri-o://73ca2fff44863b306899913da208578838945caa2a7639e95b4d983d233172b5" gracePeriod=10 Jul 18 02:57:58.652354 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:57:58.652328 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fknfc" Jul 18 02:57:58.657277 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:57:58.657252 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lj4xp\" (UniqueName: \"kubernetes.io/projected/2e4157fe-b9c3-4231-b1ec-482dbe1064ab-kube-api-access-lj4xp\") pod \"2e4157fe-b9c3-4231-b1ec-482dbe1064ab\" (UID: \"2e4157fe-b9c3-4231-b1ec-482dbe1064ab\") " Jul 18 02:57:58.657389 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:57:58.657324 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e4157fe-b9c3-4231-b1ec-482dbe1064ab-extensions-socket-volume\") pod \"2e4157fe-b9c3-4231-b1ec-482dbe1064ab\" (UID: \"2e4157fe-b9c3-4231-b1ec-482dbe1064ab\") " Jul 18 02:57:58.657654 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:57:58.657633 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4157fe-b9c3-4231-b1ec-482dbe1064ab-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2e4157fe-b9c3-4231-b1ec-482dbe1064ab" (UID: "2e4157fe-b9c3-4231-b1ec-482dbe1064ab"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 02:57:58.659124 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:57:58.659104 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e4157fe-b9c3-4231-b1ec-482dbe1064ab-kube-api-access-lj4xp" (OuterVolumeSpecName: "kube-api-access-lj4xp") pod "2e4157fe-b9c3-4231-b1ec-482dbe1064ab" (UID: "2e4157fe-b9c3-4231-b1ec-482dbe1064ab"). InnerVolumeSpecName "kube-api-access-lj4xp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 02:57:58.757933 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:57:58.757899 2585 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e4157fe-b9c3-4231-b1ec-482dbe1064ab-extensions-socket-volume\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 02:57:58.757933 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:57:58.757928 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj4xp\" (UniqueName: \"kubernetes.io/projected/2e4157fe-b9c3-4231-b1ec-482dbe1064ab-kube-api-access-lj4xp\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 02:57:58.992307 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:57:58.992273 2585 generic.go:356] "Generic (PLEG): container finished" podID="2e4157fe-b9c3-4231-b1ec-482dbe1064ab" containerID="73ca2fff44863b306899913da208578838945caa2a7639e95b4d983d233172b5" exitCode=0 Jul 18 02:57:58.992489 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:57:58.992329 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fknfc" Jul 18 02:57:58.992489 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:57:58.992351 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fknfc" event={"ID":"2e4157fe-b9c3-4231-b1ec-482dbe1064ab","Type":"ContainerDied","Data":"73ca2fff44863b306899913da208578838945caa2a7639e95b4d983d233172b5"} Jul 18 02:57:58.992489 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:57:58.992409 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fknfc" event={"ID":"2e4157fe-b9c3-4231-b1ec-482dbe1064ab","Type":"ContainerDied","Data":"a342cb873e748066992cc53a76fc76a651633a34f917cf0dbd7d1d32f6021e10"} Jul 18 02:57:58.992489 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:57:58.992431 2585 scope.go:117] "RemoveContainer" containerID="73ca2fff44863b306899913da208578838945caa2a7639e95b4d983d233172b5" Jul 18 02:57:59.001395 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:57:59.001378 2585 scope.go:117] "RemoveContainer" containerID="73ca2fff44863b306899913da208578838945caa2a7639e95b4d983d233172b5" Jul 18 02:57:59.001663 ip-10-0-140-4 kubenswrapper[2585]: E0718 02:57:59.001641 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73ca2fff44863b306899913da208578838945caa2a7639e95b4d983d233172b5\": container with ID starting with 73ca2fff44863b306899913da208578838945caa2a7639e95b4d983d233172b5 not found: ID does not exist" containerID="73ca2fff44863b306899913da208578838945caa2a7639e95b4d983d233172b5" Jul 18 02:57:59.001702 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:57:59.001670 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73ca2fff44863b306899913da208578838945caa2a7639e95b4d983d233172b5"} err="failed to get container status \"73ca2fff44863b306899913da208578838945caa2a7639e95b4d983d233172b5\": rpc error: code = NotFound desc = could not find container \"73ca2fff44863b306899913da208578838945caa2a7639e95b4d983d233172b5\": container with ID starting with 73ca2fff44863b306899913da208578838945caa2a7639e95b4d983d233172b5 not found: ID does not exist" Jul 18 02:57:59.010613 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:57:59.010554 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fknfc"] Jul 18 02:57:59.015419 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:57:59.015396 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fknfc"] Jul 18 02:58:00.899934 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:58:00.899897 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e4157fe-b9c3-4231-b1ec-482dbe1064ab" path="/var/lib/kubelet/pods/2e4157fe-b9c3-4231-b1ec-482dbe1064ab/volumes" Jul 18 02:58:56.927689 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:58:56.927653 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-czw74"] Jul 18 02:58:56.928727 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:58:56.928688 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92623ee5-bdf3-4973-944e-ae6662d327ac" containerName="manager" Jul 18 02:58:56.928727 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:58:56.928732 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="92623ee5-bdf3-4973-944e-ae6662d327ac" containerName="manager" Jul 18 02:58:56.928909 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:58:56.928756 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e4157fe-b9c3-4231-b1ec-482dbe1064ab" containerName="manager" Jul 18 02:58:56.928909 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:58:56.928765 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e4157fe-b9c3-4231-b1ec-482dbe1064ab" containerName="manager" Jul 18 02:58:56.929004 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:58:56.928949 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e4157fe-b9c3-4231-b1ec-482dbe1064ab" containerName="manager" Jul 18 02:58:56.929004 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:58:56.928960 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="92623ee5-bdf3-4973-944e-ae6662d327ac" containerName="manager" Jul 18 02:58:56.937852 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:58:56.937829 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-czw74" Jul 18 02:58:56.942175 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:58:56.942144 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-mtc7r\"" Jul 18 02:58:56.944101 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:58:56.944073 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/873dbb75-9c87-4b8f-9b94-c94f274ab3f6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-czw74\" (UID: \"873dbb75-9c87-4b8f-9b94-c94f274ab3f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-czw74" Jul 18 02:58:56.944236 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:58:56.944160 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvmfz\" (UniqueName: \"kubernetes.io/projected/873dbb75-9c87-4b8f-9b94-c94f274ab3f6-kube-api-access-tvmfz\") pod \"kuadrant-operator-controller-manager-86878c769f-czw74\" (UID: \"873dbb75-9c87-4b8f-9b94-c94f274ab3f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-czw74" Jul 18 02:58:56.951094 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:58:56.951073 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-czw74"] Jul 18 02:58:57.044632 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:58:57.044595 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvmfz\" (UniqueName: \"kubernetes.io/projected/873dbb75-9c87-4b8f-9b94-c94f274ab3f6-kube-api-access-tvmfz\") pod \"kuadrant-operator-controller-manager-86878c769f-czw74\" (UID: \"873dbb75-9c87-4b8f-9b94-c94f274ab3f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-czw74" Jul 18 02:58:57.044800 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:58:57.044655 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/873dbb75-9c87-4b8f-9b94-c94f274ab3f6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-czw74\" (UID: \"873dbb75-9c87-4b8f-9b94-c94f274ab3f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-czw74" Jul 18 02:58:57.045025 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:58:57.045005 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/873dbb75-9c87-4b8f-9b94-c94f274ab3f6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-czw74\" (UID: \"873dbb75-9c87-4b8f-9b94-c94f274ab3f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-czw74" Jul 18 02:58:57.056609 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:58:57.056581 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvmfz\" (UniqueName: \"kubernetes.io/projected/873dbb75-9c87-4b8f-9b94-c94f274ab3f6-kube-api-access-tvmfz\") pod \"kuadrant-operator-controller-manager-86878c769f-czw74\" (UID: \"873dbb75-9c87-4b8f-9b94-c94f274ab3f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-czw74" Jul 18 02:58:57.247525 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:58:57.247494 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-czw74" Jul 18 02:58:57.387276 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:58:57.387250 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-czw74"] Jul 18 02:58:57.390387 ip-10-0-140-4 kubenswrapper[2585]: W0718 02:58:57.390349 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod873dbb75_9c87_4b8f_9b94_c94f274ab3f6.slice/crio-031e83799a46a6b92fd1756f2bf5955ffa193a653ba2e7bfd62391ab20055844 WatchSource:0}: Error finding container 031e83799a46a6b92fd1756f2bf5955ffa193a653ba2e7bfd62391ab20055844: Status 404 returned error can't find the container with id 031e83799a46a6b92fd1756f2bf5955ffa193a653ba2e7bfd62391ab20055844 Jul 18 02:58:58.265223 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:58:58.265175 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-czw74" event={"ID":"873dbb75-9c87-4b8f-9b94-c94f274ab3f6","Type":"ContainerStarted","Data":"e2195c09d621e0c145fc82e145f906b2858816df2374d11db24f9f0ec6512c44"} Jul 18 02:58:58.265609 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:58:58.265239 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-czw74" event={"ID":"873dbb75-9c87-4b8f-9b94-c94f274ab3f6","Type":"ContainerStarted","Data":"031e83799a46a6b92fd1756f2bf5955ffa193a653ba2e7bfd62391ab20055844"} Jul 18 02:58:58.265609 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:58:58.265336 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-czw74" Jul 18 02:58:58.291825 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:58:58.291772 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-czw74" podStartSLOduration=2.291753897 podStartE2EDuration="2.291753897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 02:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 02:58:58.290547505 +0000 UTC m=+1161.894195193" watchObservedRunningTime="2026-07-18 02:58:58.291753897 +0000 UTC m=+1161.895401582" Jul 18 02:59:09.271232 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:59:09.271200 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-czw74" Jul 18 02:59:36.930003 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:59:36.929970 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-krtsg_709575dc-6eae-43e1-9208-cc228795929a/console-operator/1.log" Jul 18 02:59:36.935022 ip-10-0-140-4 kubenswrapper[2585]: I0718 02:59:36.934994 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-krtsg_709575dc-6eae-43e1-9208-cc228795929a/console-operator/1.log" Jul 18 03:04:36.958742 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:04:36.958711 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-krtsg_709575dc-6eae-43e1-9208-cc228795929a/console-operator/1.log" Jul 18 03:04:36.964588 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:04:36.964554 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-krtsg_709575dc-6eae-43e1-9208-cc228795929a/console-operator/1.log" Jul 18 03:05:19.105110 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:19.105037 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d"] Jul 18 03:05:19.107941 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:19.107917 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d" Jul 18 03:05:19.110764 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:19.110739 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-43e0ec51-serving-cert\"" Jul 18 03:05:19.120704 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:19.120680 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d"] Jul 18 03:05:19.244570 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:19.244538 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1adeda20-8f40-4a4e-b441-ee42a9d11645-maas-api-tls\") pod \"maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d\" (UID: \"1adeda20-8f40-4a4e-b441-ee42a9d11645\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d" Jul 18 03:05:19.244737 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:19.244575 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1adeda20-8f40-4a4e-b441-ee42a9d11645-openshift-service-ca\") pod \"maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d\" (UID: \"1adeda20-8f40-4a4e-b441-ee42a9d11645\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d" Jul 18 03:05:19.244737 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:19.244609 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrfcp\" (UniqueName: \"kubernetes.io/projected/1adeda20-8f40-4a4e-b441-ee42a9d11645-kube-api-access-lrfcp\") pod \"maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d\" (UID: \"1adeda20-8f40-4a4e-b441-ee42a9d11645\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d" Jul 18 03:05:19.345640 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:19.345605 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1adeda20-8f40-4a4e-b441-ee42a9d11645-maas-api-tls\") pod \"maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d\" (UID: \"1adeda20-8f40-4a4e-b441-ee42a9d11645\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d" Jul 18 03:05:19.345640 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:19.345644 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1adeda20-8f40-4a4e-b441-ee42a9d11645-openshift-service-ca\") pod \"maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d\" (UID: \"1adeda20-8f40-4a4e-b441-ee42a9d11645\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d" Jul 18 03:05:19.345860 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:19.345759 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrfcp\" (UniqueName: \"kubernetes.io/projected/1adeda20-8f40-4a4e-b441-ee42a9d11645-kube-api-access-lrfcp\") pod \"maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d\" (UID: \"1adeda20-8f40-4a4e-b441-ee42a9d11645\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d" Jul 18 03:05:19.346269 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:19.346250 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1adeda20-8f40-4a4e-b441-ee42a9d11645-openshift-service-ca\") pod \"maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d\" (UID: \"1adeda20-8f40-4a4e-b441-ee42a9d11645\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d" Jul 18 03:05:19.347964 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:19.347944 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1adeda20-8f40-4a4e-b441-ee42a9d11645-maas-api-tls\") pod \"maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d\" (UID: \"1adeda20-8f40-4a4e-b441-ee42a9d11645\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d" Jul 18 03:05:19.354340 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:19.354318 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrfcp\" (UniqueName: \"kubernetes.io/projected/1adeda20-8f40-4a4e-b441-ee42a9d11645-kube-api-access-lrfcp\") pod \"maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d\" (UID: \"1adeda20-8f40-4a4e-b441-ee42a9d11645\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d" Jul 18 03:05:19.421300 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:19.421236 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d" Jul 18 03:05:19.556113 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:19.556087 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d"] Jul 18 03:05:19.558080 ip-10-0-140-4 kubenswrapper[2585]: W0718 03:05:19.558051 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adeda20_8f40_4a4e_b441_ee42a9d11645.slice/crio-b73b20fd9617dbb4321ad054f7e030ba7c72327ab92a98ce4375903de26c9f98 WatchSource:0}: Error finding container b73b20fd9617dbb4321ad054f7e030ba7c72327ab92a98ce4375903de26c9f98: Status 404 returned error can't find the container with id b73b20fd9617dbb4321ad054f7e030ba7c72327ab92a98ce4375903de26c9f98 Jul 18 03:05:19.559536 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:19.559520 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 03:05:20.052441 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:20.052407 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d" event={"ID":"1adeda20-8f40-4a4e-b441-ee42a9d11645","Type":"ContainerStarted","Data":"b73b20fd9617dbb4321ad054f7e030ba7c72327ab92a98ce4375903de26c9f98"} Jul 18 03:05:21.058328 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:21.058293 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d" event={"ID":"1adeda20-8f40-4a4e-b441-ee42a9d11645","Type":"ContainerStarted","Data":"13e92064a3ada9470ecc430e1a4c7f5c09f9dcef9e347864ece4c3a535b87d87"} Jul 18 03:05:21.058725 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:21.058356 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d" Jul 18 03:05:21.078520 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:21.078473 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d" podStartSLOduration=1.493670299 podStartE2EDuration="2.078458979s" totalImagesPullingTime="584.78868ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 03:05:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 03:05:21.076253396 +0000 UTC m=+1544.679901084" watchObservedRunningTime="2026-07-18 03:05:21.078458979 +0000 UTC m=+1544.682106666" Jul 18 03:05:27.068473 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:27.068441 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d" Jul 18 03:05:55.148421 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:55.148386 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d"] Jul 18 03:05:55.148999 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:55.148659 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d" podUID="1adeda20-8f40-4a4e-b441-ee42a9d11645" containerName="maas-api" containerID="cri-o://13e92064a3ada9470ecc430e1a4c7f5c09f9dcef9e347864ece4c3a535b87d87" gracePeriod=30 Jul 18 03:05:55.399163 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:55.399111 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d" Jul 18 03:05:55.552511 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:55.552475 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1adeda20-8f40-4a4e-b441-ee42a9d11645-maas-api-tls\") pod \"1adeda20-8f40-4a4e-b441-ee42a9d11645\" (UID: \"1adeda20-8f40-4a4e-b441-ee42a9d11645\") " Jul 18 03:05:55.552681 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:55.552547 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1adeda20-8f40-4a4e-b441-ee42a9d11645-openshift-service-ca\") pod \"1adeda20-8f40-4a4e-b441-ee42a9d11645\" (UID: \"1adeda20-8f40-4a4e-b441-ee42a9d11645\") " Jul 18 03:05:55.552681 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:55.552612 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrfcp\" (UniqueName: \"kubernetes.io/projected/1adeda20-8f40-4a4e-b441-ee42a9d11645-kube-api-access-lrfcp\") pod \"1adeda20-8f40-4a4e-b441-ee42a9d11645\" (UID: \"1adeda20-8f40-4a4e-b441-ee42a9d11645\") " Jul 18 03:05:55.552965 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:55.552942 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1adeda20-8f40-4a4e-b441-ee42a9d11645-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1adeda20-8f40-4a4e-b441-ee42a9d11645" (UID: "1adeda20-8f40-4a4e-b441-ee42a9d11645"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 03:05:55.554703 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:55.554678 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1adeda20-8f40-4a4e-b441-ee42a9d11645-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1adeda20-8f40-4a4e-b441-ee42a9d11645" (UID: "1adeda20-8f40-4a4e-b441-ee42a9d11645"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 03:05:55.554830 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:55.554809 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1adeda20-8f40-4a4e-b441-ee42a9d11645-kube-api-access-lrfcp" (OuterVolumeSpecName: "kube-api-access-lrfcp") pod "1adeda20-8f40-4a4e-b441-ee42a9d11645" (UID: "1adeda20-8f40-4a4e-b441-ee42a9d11645"). InnerVolumeSpecName "kube-api-access-lrfcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 03:05:55.653512 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:55.653417 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1adeda20-8f40-4a4e-b441-ee42a9d11645-maas-api-tls\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:05:55.653512 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:55.653456 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1adeda20-8f40-4a4e-b441-ee42a9d11645-openshift-service-ca\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:05:55.653512 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:55.653469 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrfcp\" (UniqueName: \"kubernetes.io/projected/1adeda20-8f40-4a4e-b441-ee42a9d11645-kube-api-access-lrfcp\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:05:56.226596 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:56.226556 2585 generic.go:356] "Generic (PLEG): container finished" podID="1adeda20-8f40-4a4e-b441-ee42a9d11645" containerID="13e92064a3ada9470ecc430e1a4c7f5c09f9dcef9e347864ece4c3a535b87d87" exitCode=0 Jul 18 03:05:56.226596 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:56.226601 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d" Jul 18 03:05:56.227092 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:56.226623 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d" event={"ID":"1adeda20-8f40-4a4e-b441-ee42a9d11645","Type":"ContainerDied","Data":"13e92064a3ada9470ecc430e1a4c7f5c09f9dcef9e347864ece4c3a535b87d87"} Jul 18 03:05:56.227092 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:56.226676 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d" event={"ID":"1adeda20-8f40-4a4e-b441-ee42a9d11645","Type":"ContainerDied","Data":"b73b20fd9617dbb4321ad054f7e030ba7c72327ab92a98ce4375903de26c9f98"} Jul 18 03:05:56.227092 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:56.226708 2585 scope.go:117] "RemoveContainer" containerID="13e92064a3ada9470ecc430e1a4c7f5c09f9dcef9e347864ece4c3a535b87d87" Jul 18 03:05:56.235802 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:56.235784 2585 scope.go:117] "RemoveContainer" containerID="13e92064a3ada9470ecc430e1a4c7f5c09f9dcef9e347864ece4c3a535b87d87" Jul 18 03:05:56.236072 ip-10-0-140-4 kubenswrapper[2585]: E0718 03:05:56.236052 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13e92064a3ada9470ecc430e1a4c7f5c09f9dcef9e347864ece4c3a535b87d87\": container with ID starting with 13e92064a3ada9470ecc430e1a4c7f5c09f9dcef9e347864ece4c3a535b87d87 not found: ID does not exist" containerID="13e92064a3ada9470ecc430e1a4c7f5c09f9dcef9e347864ece4c3a535b87d87" Jul 18 03:05:56.236120 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:56.236081 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13e92064a3ada9470ecc430e1a4c7f5c09f9dcef9e347864ece4c3a535b87d87"} err="failed to get container status \"13e92064a3ada9470ecc430e1a4c7f5c09f9dcef9e347864ece4c3a535b87d87\": rpc error: code = NotFound desc = could not find container \"13e92064a3ada9470ecc430e1a4c7f5c09f9dcef9e347864ece4c3a535b87d87\": container with ID starting with 13e92064a3ada9470ecc430e1a4c7f5c09f9dcef9e347864ece4c3a535b87d87 not found: ID does not exist" Jul 18 03:05:56.247902 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:56.247878 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d"] Jul 18 03:05:56.251332 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:56.251309 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-43e0ec51-75fcc5f678-h8t6d"] Jul 18 03:05:56.900515 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:05:56.900481 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1adeda20-8f40-4a4e-b441-ee42a9d11645" path="/var/lib/kubelet/pods/1adeda20-8f40-4a4e-b441-ee42a9d11645/volumes" Jul 18 03:06:03.590414 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:03.590375 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj"] Jul 18 03:06:03.591345 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:03.591319 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1adeda20-8f40-4a4e-b441-ee42a9d11645" containerName="maas-api" Jul 18 03:06:03.591463 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:03.591349 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1adeda20-8f40-4a4e-b441-ee42a9d11645" containerName="maas-api" Jul 18 03:06:03.591539 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:03.591526 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1adeda20-8f40-4a4e-b441-ee42a9d11645" containerName="maas-api" Jul 18 03:06:03.597248 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:03.597221 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj" Jul 18 03:06:03.600525 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:03.600499 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-a70f0b50-serving-cert\"" Jul 18 03:06:03.602233 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:03.602211 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj"] Jul 18 03:06:03.723570 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:03.723534 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzxs7\" (UniqueName: \"kubernetes.io/projected/b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a-kube-api-access-wzxs7\") pod \"maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj\" (UID: \"b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj" Jul 18 03:06:03.723741 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:03.723608 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a-maas-api-tls\") pod \"maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj\" (UID: \"b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj" Jul 18 03:06:03.723741 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:03.723697 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a-openshift-service-ca\") pod \"maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj\" (UID: \"b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj" Jul 18 03:06:03.824806 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:03.824761 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzxs7\" (UniqueName: \"kubernetes.io/projected/b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a-kube-api-access-wzxs7\") pod \"maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj\" (UID: \"b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj" Jul 18 03:06:03.824993 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:03.824814 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a-maas-api-tls\") pod \"maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj\" (UID: \"b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj" Jul 18 03:06:03.824993 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:03.824888 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a-openshift-service-ca\") pod \"maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj\" (UID: \"b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj" Jul 18 03:06:03.825560 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:03.825537 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a-openshift-service-ca\") pod \"maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj\" (UID: \"b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj" Jul 18 03:06:03.827254 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:03.827231 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a-maas-api-tls\") pod \"maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj\" (UID: \"b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj" Jul 18 03:06:03.833173 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:03.833148 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzxs7\" (UniqueName: \"kubernetes.io/projected/b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a-kube-api-access-wzxs7\") pod \"maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj\" (UID: \"b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj" Jul 18 03:06:03.909813 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:03.909739 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj" Jul 18 03:06:04.052170 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:04.052146 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj"] Jul 18 03:06:04.054060 ip-10-0-140-4 kubenswrapper[2585]: W0718 03:06:04.054031 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ab9f00_87ff_4b2e_93e5_c00f91d9b35a.slice/crio-110d056ab48e1d91402fe60b75b5940a6b47d14d7c028699bb80896b2644e5a3 WatchSource:0}: Error finding container 110d056ab48e1d91402fe60b75b5940a6b47d14d7c028699bb80896b2644e5a3: Status 404 returned error can't find the container with id 110d056ab48e1d91402fe60b75b5940a6b47d14d7c028699bb80896b2644e5a3 Jul 18 03:06:04.266320 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:04.266281 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj" event={"ID":"b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a","Type":"ContainerStarted","Data":"110d056ab48e1d91402fe60b75b5940a6b47d14d7c028699bb80896b2644e5a3"} Jul 18 03:06:05.272557 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:05.272522 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj" event={"ID":"b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a","Type":"ContainerStarted","Data":"e38afec64a6b1b391591d6b7eb70ef50bf6a4a25371f55f62aa024d0be0ad61c"} Jul 18 03:06:05.272972 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:05.272626 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj" Jul 18 03:06:05.290384 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:05.290335 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj" podStartSLOduration=1.889492961 podStartE2EDuration="2.290320118s" totalImagesPullingTime="400.827157ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 03:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 03:06:05.288707485 +0000 UTC m=+1588.892355175" watchObservedRunningTime="2026-07-18 03:06:05.290320118 +0000 UTC m=+1588.893967806" Jul 18 03:06:11.283571 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:11.283542 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj" Jul 18 03:06:38.793862 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:38.793826 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj"] Jul 18 03:06:38.794292 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:38.794056 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj" podUID="b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a" containerName="maas-api" containerID="cri-o://e38afec64a6b1b391591d6b7eb70ef50bf6a4a25371f55f62aa024d0be0ad61c" gracePeriod=30 Jul 18 03:06:39.057009 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:39.056950 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj" Jul 18 03:06:39.123541 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:39.123510 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a-maas-api-tls\") pod \"b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a\" (UID: \"b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a\") " Jul 18 03:06:39.123695 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:39.123632 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzxs7\" (UniqueName: \"kubernetes.io/projected/b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a-kube-api-access-wzxs7\") pod \"b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a\" (UID: \"b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a\") " Jul 18 03:06:39.123695 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:39.123669 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a-openshift-service-ca\") pod \"b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a\" (UID: \"b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a\") " Jul 18 03:06:39.124065 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:39.124036 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a" (UID: "b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 03:06:39.125603 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:39.125582 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a" (UID: "b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 03:06:39.125603 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:39.125586 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a-kube-api-access-wzxs7" (OuterVolumeSpecName: "kube-api-access-wzxs7") pod "b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a" (UID: "b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a"). InnerVolumeSpecName "kube-api-access-wzxs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 03:06:39.224600 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:39.224556 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzxs7\" (UniqueName: \"kubernetes.io/projected/b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a-kube-api-access-wzxs7\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:06:39.224600 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:39.224592 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a-openshift-service-ca\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:06:39.224600 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:39.224602 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a-maas-api-tls\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:06:39.439084 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:39.438985 2585 generic.go:356] "Generic (PLEG): container finished" podID="b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a" containerID="e38afec64a6b1b391591d6b7eb70ef50bf6a4a25371f55f62aa024d0be0ad61c" exitCode=0 Jul 18 03:06:39.439084 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:39.439026 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj" Jul 18 03:06:39.439084 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:39.439060 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj" event={"ID":"b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a","Type":"ContainerDied","Data":"e38afec64a6b1b391591d6b7eb70ef50bf6a4a25371f55f62aa024d0be0ad61c"} Jul 18 03:06:39.439401 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:39.439115 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj" event={"ID":"b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a","Type":"ContainerDied","Data":"110d056ab48e1d91402fe60b75b5940a6b47d14d7c028699bb80896b2644e5a3"} Jul 18 03:06:39.439401 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:39.439142 2585 scope.go:117] "RemoveContainer" containerID="e38afec64a6b1b391591d6b7eb70ef50bf6a4a25371f55f62aa024d0be0ad61c" Jul 18 03:06:39.450677 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:39.449490 2585 scope.go:117] "RemoveContainer" containerID="e38afec64a6b1b391591d6b7eb70ef50bf6a4a25371f55f62aa024d0be0ad61c" Jul 18 03:06:39.451091 ip-10-0-140-4 kubenswrapper[2585]: E0718 03:06:39.451061 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e38afec64a6b1b391591d6b7eb70ef50bf6a4a25371f55f62aa024d0be0ad61c\": container with ID starting with e38afec64a6b1b391591d6b7eb70ef50bf6a4a25371f55f62aa024d0be0ad61c not found: ID does not exist" containerID="e38afec64a6b1b391591d6b7eb70ef50bf6a4a25371f55f62aa024d0be0ad61c" Jul 18 03:06:39.451238 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:39.451103 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e38afec64a6b1b391591d6b7eb70ef50bf6a4a25371f55f62aa024d0be0ad61c"} err="failed to get container status \"e38afec64a6b1b391591d6b7eb70ef50bf6a4a25371f55f62aa024d0be0ad61c\": rpc error: code = NotFound desc = could not find container \"e38afec64a6b1b391591d6b7eb70ef50bf6a4a25371f55f62aa024d0be0ad61c\": container with ID starting with e38afec64a6b1b391591d6b7eb70ef50bf6a4a25371f55f62aa024d0be0ad61c not found: ID does not exist" Jul 18 03:06:39.461238 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:39.461216 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj"] Jul 18 03:06:39.463027 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:39.463004 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a70f0b50-8489d78b66-7mbqj"] Jul 18 03:06:40.900525 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:40.900495 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a" path="/var/lib/kubelet/pods/b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a/volumes" Jul 18 03:06:46.685988 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:46.685947 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l"] Jul 18 03:06:46.686461 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:46.686354 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a" containerName="maas-api" Jul 18 03:06:46.686461 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:46.686366 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a" containerName="maas-api" Jul 18 03:06:46.686461 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:46.686435 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4ab9f00-87ff-4b2e-93e5-c00f91d9b35a" containerName="maas-api" Jul 18 03:06:46.690668 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:46.690650 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l" Jul 18 03:06:46.693219 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:46.693197 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-99930a7a-serving-cert\"" Jul 18 03:06:46.700352 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:46.700322 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l"] Jul 18 03:06:46.790889 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:46.790860 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd781e6-298a-4bf0-94a2-56b2d05b39e6-openshift-service-ca\") pod \"maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l\" (UID: \"8bd781e6-298a-4bf0-94a2-56b2d05b39e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l" Jul 18 03:06:46.791039 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:46.790949 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwwc6\" (UniqueName: \"kubernetes.io/projected/8bd781e6-298a-4bf0-94a2-56b2d05b39e6-kube-api-access-rwwc6\") pod \"maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l\" (UID: \"8bd781e6-298a-4bf0-94a2-56b2d05b39e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l" Jul 18 03:06:46.791039 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:46.790987 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd781e6-298a-4bf0-94a2-56b2d05b39e6-maas-api-tls\") pod \"maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l\" (UID: \"8bd781e6-298a-4bf0-94a2-56b2d05b39e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l" Jul 18 03:06:46.892307 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:46.892275 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwwc6\" (UniqueName: \"kubernetes.io/projected/8bd781e6-298a-4bf0-94a2-56b2d05b39e6-kube-api-access-rwwc6\") pod \"maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l\" (UID: \"8bd781e6-298a-4bf0-94a2-56b2d05b39e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l" Jul 18 03:06:46.892454 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:46.892318 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd781e6-298a-4bf0-94a2-56b2d05b39e6-maas-api-tls\") pod \"maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l\" (UID: \"8bd781e6-298a-4bf0-94a2-56b2d05b39e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l" Jul 18 03:06:46.892454 ip-10-0-140-4 kubenswrapper[2585]: E0718 03:06:46.892418 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-99930a7a-serving-cert: secret "maas-api-e2e-ait-99930a7a-serving-cert" not found Jul 18 03:06:46.892454 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:46.892442 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd781e6-298a-4bf0-94a2-56b2d05b39e6-openshift-service-ca\") pod \"maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l\" (UID: \"8bd781e6-298a-4bf0-94a2-56b2d05b39e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l" Jul 18 03:06:46.892562 ip-10-0-140-4 kubenswrapper[2585]: E0718 03:06:46.892482 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bd781e6-298a-4bf0-94a2-56b2d05b39e6-maas-api-tls podName:8bd781e6-298a-4bf0-94a2-56b2d05b39e6 nodeName:}" failed. No retries permitted until 2026-07-18 03:06:47.392465654 +0000 UTC m=+1630.996113319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8bd781e6-298a-4bf0-94a2-56b2d05b39e6-maas-api-tls") pod "maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l" (UID: "8bd781e6-298a-4bf0-94a2-56b2d05b39e6") : secret "maas-api-e2e-ait-99930a7a-serving-cert" not found Jul 18 03:06:46.893050 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:46.893030 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd781e6-298a-4bf0-94a2-56b2d05b39e6-openshift-service-ca\") pod \"maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l\" (UID: \"8bd781e6-298a-4bf0-94a2-56b2d05b39e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l" Jul 18 03:06:46.900939 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:46.900916 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwwc6\" (UniqueName: \"kubernetes.io/projected/8bd781e6-298a-4bf0-94a2-56b2d05b39e6-kube-api-access-rwwc6\") pod \"maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l\" (UID: \"8bd781e6-298a-4bf0-94a2-56b2d05b39e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l" Jul 18 03:06:47.397003 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:47.396968 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd781e6-298a-4bf0-94a2-56b2d05b39e6-maas-api-tls\") pod \"maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l\" (UID: \"8bd781e6-298a-4bf0-94a2-56b2d05b39e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l" Jul 18 03:06:47.399481 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:47.399457 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd781e6-298a-4bf0-94a2-56b2d05b39e6-maas-api-tls\") pod \"maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l\" (UID: \"8bd781e6-298a-4bf0-94a2-56b2d05b39e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l" Jul 18 03:06:47.601852 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:47.601815 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l" Jul 18 03:06:47.941470 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:47.941439 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l"] Jul 18 03:06:47.942384 ip-10-0-140-4 kubenswrapper[2585]: W0718 03:06:47.942350 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd781e6_298a_4bf0_94a2_56b2d05b39e6.slice/crio-c8e9a27451880eddf0ec7a7cb4e1f7c5dcbd8ce992b39d1012053eb900522fb6 WatchSource:0}: Error finding container c8e9a27451880eddf0ec7a7cb4e1f7c5dcbd8ce992b39d1012053eb900522fb6: Status 404 returned error can't find the container with id c8e9a27451880eddf0ec7a7cb4e1f7c5dcbd8ce992b39d1012053eb900522fb6 Jul 18 03:06:48.486696 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:48.486658 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l" event={"ID":"8bd781e6-298a-4bf0-94a2-56b2d05b39e6","Type":"ContainerStarted","Data":"c8e9a27451880eddf0ec7a7cb4e1f7c5dcbd8ce992b39d1012053eb900522fb6"} Jul 18 03:06:49.493089 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:49.493052 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l" event={"ID":"8bd781e6-298a-4bf0-94a2-56b2d05b39e6","Type":"ContainerStarted","Data":"a6308b30e8b665936385f29788623bc2fb1e6e4a2b5a4347c9608c4ce1e55674"} Jul 18 03:06:49.493597 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:49.493161 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l" Jul 18 03:06:49.510911 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:49.510868 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l" podStartSLOduration=2.852286104 podStartE2EDuration="3.510854702s" totalImagesPullingTime="658.568598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 03:06:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 03:06:49.509646342 +0000 UTC m=+1633.113294029" watchObservedRunningTime="2026-07-18 03:06:49.510854702 +0000 UTC m=+1633.114502408" Jul 18 03:06:55.505369 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:06:55.505336 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l" Jul 18 03:07:22.719610 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:22.719572 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l"] Jul 18 03:07:22.720323 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:22.719815 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l" podUID="8bd781e6-298a-4bf0-94a2-56b2d05b39e6" containerName="maas-api" containerID="cri-o://a6308b30e8b665936385f29788623bc2fb1e6e4a2b5a4347c9608c4ce1e55674" gracePeriod=30 Jul 18 03:07:22.968471 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:22.968444 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l" Jul 18 03:07:23.094845 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:23.094761 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd781e6-298a-4bf0-94a2-56b2d05b39e6-openshift-service-ca\") pod \"8bd781e6-298a-4bf0-94a2-56b2d05b39e6\" (UID: \"8bd781e6-298a-4bf0-94a2-56b2d05b39e6\") " Jul 18 03:07:23.094845 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:23.094815 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd781e6-298a-4bf0-94a2-56b2d05b39e6-maas-api-tls\") pod \"8bd781e6-298a-4bf0-94a2-56b2d05b39e6\" (UID: \"8bd781e6-298a-4bf0-94a2-56b2d05b39e6\") " Jul 18 03:07:23.095084 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:23.094877 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwwc6\" (UniqueName: \"kubernetes.io/projected/8bd781e6-298a-4bf0-94a2-56b2d05b39e6-kube-api-access-rwwc6\") pod \"8bd781e6-298a-4bf0-94a2-56b2d05b39e6\" (UID: \"8bd781e6-298a-4bf0-94a2-56b2d05b39e6\") " Jul 18 03:07:23.095126 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:23.095095 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bd781e6-298a-4bf0-94a2-56b2d05b39e6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8bd781e6-298a-4bf0-94a2-56b2d05b39e6" (UID: "8bd781e6-298a-4bf0-94a2-56b2d05b39e6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 03:07:23.096903 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:23.096879 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd781e6-298a-4bf0-94a2-56b2d05b39e6-kube-api-access-rwwc6" (OuterVolumeSpecName: "kube-api-access-rwwc6") pod "8bd781e6-298a-4bf0-94a2-56b2d05b39e6" (UID: "8bd781e6-298a-4bf0-94a2-56b2d05b39e6"). InnerVolumeSpecName "kube-api-access-rwwc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 03:07:23.096903 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:23.096882 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bd781e6-298a-4bf0-94a2-56b2d05b39e6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8bd781e6-298a-4bf0-94a2-56b2d05b39e6" (UID: "8bd781e6-298a-4bf0-94a2-56b2d05b39e6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 03:07:23.196487 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:23.196455 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwwc6\" (UniqueName: \"kubernetes.io/projected/8bd781e6-298a-4bf0-94a2-56b2d05b39e6-kube-api-access-rwwc6\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:07:23.196487 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:23.196482 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd781e6-298a-4bf0-94a2-56b2d05b39e6-openshift-service-ca\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:07:23.196487 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:23.196492 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd781e6-298a-4bf0-94a2-56b2d05b39e6-maas-api-tls\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:07:23.656742 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:23.656701 2585 generic.go:356] "Generic (PLEG): container finished" podID="8bd781e6-298a-4bf0-94a2-56b2d05b39e6" containerID="a6308b30e8b665936385f29788623bc2fb1e6e4a2b5a4347c9608c4ce1e55674" exitCode=0 Jul 18 03:07:23.656942 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:23.656748 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l" Jul 18 03:07:23.656942 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:23.656794 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l" event={"ID":"8bd781e6-298a-4bf0-94a2-56b2d05b39e6","Type":"ContainerDied","Data":"a6308b30e8b665936385f29788623bc2fb1e6e4a2b5a4347c9608c4ce1e55674"} Jul 18 03:07:23.656942 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:23.656854 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l" event={"ID":"8bd781e6-298a-4bf0-94a2-56b2d05b39e6","Type":"ContainerDied","Data":"c8e9a27451880eddf0ec7a7cb4e1f7c5dcbd8ce992b39d1012053eb900522fb6"} Jul 18 03:07:23.656942 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:23.656885 2585 scope.go:117] "RemoveContainer" containerID="a6308b30e8b665936385f29788623bc2fb1e6e4a2b5a4347c9608c4ce1e55674" Jul 18 03:07:23.665982 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:23.665960 2585 scope.go:117] "RemoveContainer" containerID="a6308b30e8b665936385f29788623bc2fb1e6e4a2b5a4347c9608c4ce1e55674" Jul 18 03:07:23.666222 ip-10-0-140-4 kubenswrapper[2585]: E0718 03:07:23.666205 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6308b30e8b665936385f29788623bc2fb1e6e4a2b5a4347c9608c4ce1e55674\": container with ID starting with a6308b30e8b665936385f29788623bc2fb1e6e4a2b5a4347c9608c4ce1e55674 not found: ID does not exist" containerID="a6308b30e8b665936385f29788623bc2fb1e6e4a2b5a4347c9608c4ce1e55674" Jul 18 03:07:23.666300 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:23.666228 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6308b30e8b665936385f29788623bc2fb1e6e4a2b5a4347c9608c4ce1e55674"} err="failed to get container status \"a6308b30e8b665936385f29788623bc2fb1e6e4a2b5a4347c9608c4ce1e55674\": rpc error: code = NotFound desc = could not find container \"a6308b30e8b665936385f29788623bc2fb1e6e4a2b5a4347c9608c4ce1e55674\": container with ID starting with a6308b30e8b665936385f29788623bc2fb1e6e4a2b5a4347c9608c4ce1e55674 not found: ID does not exist" Jul 18 03:07:23.678303 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:23.678281 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l"] Jul 18 03:07:23.682313 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:23.682293 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-99930a7a-75c9c8bf69-sdm8l"] Jul 18 03:07:24.900057 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:24.900023 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bd781e6-298a-4bf0-94a2-56b2d05b39e6" path="/var/lib/kubelet/pods/8bd781e6-298a-4bf0-94a2-56b2d05b39e6/volumes" Jul 18 03:07:31.992675 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:31.992644 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h"] Jul 18 03:07:31.993170 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:31.993019 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bd781e6-298a-4bf0-94a2-56b2d05b39e6" containerName="maas-api" Jul 18 03:07:31.993170 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:31.993031 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bd781e6-298a-4bf0-94a2-56b2d05b39e6" containerName="maas-api" Jul 18 03:07:31.993170 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:31.993100 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bd781e6-298a-4bf0-94a2-56b2d05b39e6" containerName="maas-api" Jul 18 03:07:31.997505 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:31.997485 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h" Jul 18 03:07:31.999949 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:31.999926 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-efca33f7-serving-cert\"" Jul 18 03:07:32.010006 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:32.009979 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h"] Jul 18 03:07:32.078634 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:32.078595 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b989508-6e04-4fbb-a840-4310c8338c40-openshift-service-ca\") pod \"maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h\" (UID: \"3b989508-6e04-4fbb-a840-4310c8338c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h" Jul 18 03:07:32.078803 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:32.078671 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b989508-6e04-4fbb-a840-4310c8338c40-maas-api-tls\") pod \"maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h\" (UID: \"3b989508-6e04-4fbb-a840-4310c8338c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h" Jul 18 03:07:32.078803 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:32.078704 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqw26\" (UniqueName: \"kubernetes.io/projected/3b989508-6e04-4fbb-a840-4310c8338c40-kube-api-access-xqw26\") pod \"maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h\" (UID: \"3b989508-6e04-4fbb-a840-4310c8338c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h" Jul 18 03:07:32.179830 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:32.179799 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b989508-6e04-4fbb-a840-4310c8338c40-maas-api-tls\") pod \"maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h\" (UID: \"3b989508-6e04-4fbb-a840-4310c8338c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h" Jul 18 03:07:32.180007 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:32.179844 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqw26\" (UniqueName: \"kubernetes.io/projected/3b989508-6e04-4fbb-a840-4310c8338c40-kube-api-access-xqw26\") pod \"maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h\" (UID: \"3b989508-6e04-4fbb-a840-4310c8338c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h" Jul 18 03:07:32.180007 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:32.179922 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b989508-6e04-4fbb-a840-4310c8338c40-openshift-service-ca\") pod \"maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h\" (UID: \"3b989508-6e04-4fbb-a840-4310c8338c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h" Jul 18 03:07:32.180658 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:32.180631 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b989508-6e04-4fbb-a840-4310c8338c40-openshift-service-ca\") pod \"maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h\" (UID: \"3b989508-6e04-4fbb-a840-4310c8338c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h" Jul 18 03:07:32.182302 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:32.182274 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b989508-6e04-4fbb-a840-4310c8338c40-maas-api-tls\") pod \"maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h\" (UID: \"3b989508-6e04-4fbb-a840-4310c8338c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h" Jul 18 03:07:32.191083 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:32.191057 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqw26\" (UniqueName: \"kubernetes.io/projected/3b989508-6e04-4fbb-a840-4310c8338c40-kube-api-access-xqw26\") pod \"maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h\" (UID: \"3b989508-6e04-4fbb-a840-4310c8338c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h" Jul 18 03:07:32.317281 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:32.317166 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h" Jul 18 03:07:32.455504 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:32.455473 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h"] Jul 18 03:07:32.457697 ip-10-0-140-4 kubenswrapper[2585]: W0718 03:07:32.457660 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b989508_6e04_4fbb_a840_4310c8338c40.slice/crio-2102840b0fab9e64562ccd3e3d88cb7eea29e5a816110a8a1932a0d9f0b248ac WatchSource:0}: Error finding container 2102840b0fab9e64562ccd3e3d88cb7eea29e5a816110a8a1932a0d9f0b248ac: Status 404 returned error can't find the container with id 2102840b0fab9e64562ccd3e3d88cb7eea29e5a816110a8a1932a0d9f0b248ac Jul 18 03:07:32.699760 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:32.699669 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h" event={"ID":"3b989508-6e04-4fbb-a840-4310c8338c40","Type":"ContainerStarted","Data":"2102840b0fab9e64562ccd3e3d88cb7eea29e5a816110a8a1932a0d9f0b248ac"} Jul 18 03:07:33.705869 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:33.705836 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h" event={"ID":"3b989508-6e04-4fbb-a840-4310c8338c40","Type":"ContainerStarted","Data":"30804c34ec2b22bc088fbc53ed98fbd6ce1edaa551fcf831a71d192e528d9a67"} Jul 18 03:07:33.706272 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:33.705959 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h" Jul 18 03:07:39.716882 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:39.716843 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h" Jul 18 03:07:39.738351 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:07:39.735399 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h" podStartSLOduration=8.248912896 podStartE2EDuration="8.735375714s" totalImagesPullingTime="486.462818ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 03:07:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 03:07:33.746745745 +0000 UTC m=+1677.350393433" watchObservedRunningTime="2026-07-18 03:07:39.735375714 +0000 UTC m=+1683.339023462" Jul 18 03:08:07.245395 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:08:07.245305 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h"] Jul 18 03:08:07.245952 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:08:07.245604 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h" podUID="3b989508-6e04-4fbb-a840-4310c8338c40" containerName="maas-api" containerID="cri-o://30804c34ec2b22bc088fbc53ed98fbd6ce1edaa551fcf831a71d192e528d9a67" gracePeriod=30 Jul 18 03:08:07.500130 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:08:07.500075 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h" Jul 18 03:08:07.592903 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:08:07.592867 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqw26\" (UniqueName: \"kubernetes.io/projected/3b989508-6e04-4fbb-a840-4310c8338c40-kube-api-access-xqw26\") pod \"3b989508-6e04-4fbb-a840-4310c8338c40\" (UID: \"3b989508-6e04-4fbb-a840-4310c8338c40\") " Jul 18 03:08:07.593082 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:08:07.592945 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b989508-6e04-4fbb-a840-4310c8338c40-openshift-service-ca\") pod \"3b989508-6e04-4fbb-a840-4310c8338c40\" (UID: \"3b989508-6e04-4fbb-a840-4310c8338c40\") " Jul 18 03:08:07.593082 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:08:07.593037 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b989508-6e04-4fbb-a840-4310c8338c40-maas-api-tls\") pod \"3b989508-6e04-4fbb-a840-4310c8338c40\" (UID: \"3b989508-6e04-4fbb-a840-4310c8338c40\") " Jul 18 03:08:07.593232 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:08:07.593207 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b989508-6e04-4fbb-a840-4310c8338c40-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3b989508-6e04-4fbb-a840-4310c8338c40" (UID: "3b989508-6e04-4fbb-a840-4310c8338c40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 03:08:07.593356 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:08:07.593339 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b989508-6e04-4fbb-a840-4310c8338c40-openshift-service-ca\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:08:07.594988 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:08:07.594965 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b989508-6e04-4fbb-a840-4310c8338c40-kube-api-access-xqw26" (OuterVolumeSpecName: "kube-api-access-xqw26") pod "3b989508-6e04-4fbb-a840-4310c8338c40" (UID: "3b989508-6e04-4fbb-a840-4310c8338c40"). InnerVolumeSpecName "kube-api-access-xqw26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 03:08:07.595067 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:08:07.594991 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b989508-6e04-4fbb-a840-4310c8338c40-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3b989508-6e04-4fbb-a840-4310c8338c40" (UID: "3b989508-6e04-4fbb-a840-4310c8338c40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 03:08:07.694203 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:08:07.694148 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b989508-6e04-4fbb-a840-4310c8338c40-maas-api-tls\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:08:07.694203 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:08:07.694176 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqw26\" (UniqueName: \"kubernetes.io/projected/3b989508-6e04-4fbb-a840-4310c8338c40-kube-api-access-xqw26\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:08:07.887529 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:08:07.887450 2585 generic.go:356] "Generic (PLEG): container finished" podID="3b989508-6e04-4fbb-a840-4310c8338c40" containerID="30804c34ec2b22bc088fbc53ed98fbd6ce1edaa551fcf831a71d192e528d9a67" exitCode=0 Jul 18 03:08:07.887529 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:08:07.887498 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h" Jul 18 03:08:07.887529 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:08:07.887517 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h" event={"ID":"3b989508-6e04-4fbb-a840-4310c8338c40","Type":"ContainerDied","Data":"30804c34ec2b22bc088fbc53ed98fbd6ce1edaa551fcf831a71d192e528d9a67"} Jul 18 03:08:07.887779 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:08:07.887561 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h" event={"ID":"3b989508-6e04-4fbb-a840-4310c8338c40","Type":"ContainerDied","Data":"2102840b0fab9e64562ccd3e3d88cb7eea29e5a816110a8a1932a0d9f0b248ac"} Jul 18 03:08:07.887779 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:08:07.887583 2585 scope.go:117] "RemoveContainer" containerID="30804c34ec2b22bc088fbc53ed98fbd6ce1edaa551fcf831a71d192e528d9a67" Jul 18 03:08:07.897318 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:08:07.897296 2585 scope.go:117] "RemoveContainer" containerID="30804c34ec2b22bc088fbc53ed98fbd6ce1edaa551fcf831a71d192e528d9a67" Jul 18 03:08:07.897567 ip-10-0-140-4 kubenswrapper[2585]: E0718 03:08:07.897545 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30804c34ec2b22bc088fbc53ed98fbd6ce1edaa551fcf831a71d192e528d9a67\": container with ID starting with 30804c34ec2b22bc088fbc53ed98fbd6ce1edaa551fcf831a71d192e528d9a67 not found: ID does not exist" containerID="30804c34ec2b22bc088fbc53ed98fbd6ce1edaa551fcf831a71d192e528d9a67" Jul 18 03:08:07.897643 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:08:07.897574 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30804c34ec2b22bc088fbc53ed98fbd6ce1edaa551fcf831a71d192e528d9a67"} err="failed to get container status \"30804c34ec2b22bc088fbc53ed98fbd6ce1edaa551fcf831a71d192e528d9a67\": rpc error: code = NotFound desc = could not find container \"30804c34ec2b22bc088fbc53ed98fbd6ce1edaa551fcf831a71d192e528d9a67\": container with ID starting with 30804c34ec2b22bc088fbc53ed98fbd6ce1edaa551fcf831a71d192e528d9a67 not found: ID does not exist" Jul 18 03:08:07.909433 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:08:07.909408 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h"] Jul 18 03:08:07.913442 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:08:07.913420 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-efca33f7-648b885cbc-zsx7h"] Jul 18 03:08:08.899768 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:08:08.899734 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b989508-6e04-4fbb-a840-4310c8338c40" path="/var/lib/kubelet/pods/3b989508-6e04-4fbb-a840-4310c8338c40/volumes" Jul 18 03:09:36.992025 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:09:36.991965 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-krtsg_709575dc-6eae-43e1-9208-cc228795929a/console-operator/1.log" Jul 18 03:09:37.002806 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:09:37.002751 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-krtsg_709575dc-6eae-43e1-9208-cc228795929a/console-operator/1.log" Jul 18 03:09:57.401905 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:09:57.401842 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1d958dfb-95d984596-sh7xb"] Jul 18 03:09:57.402691 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:09:57.402611 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b989508-6e04-4fbb-a840-4310c8338c40" containerName="maas-api" Jul 18 03:09:57.402691 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:09:57.402639 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b989508-6e04-4fbb-a840-4310c8338c40" containerName="maas-api" Jul 18 03:09:57.402818 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:09:57.402801 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b989508-6e04-4fbb-a840-4310c8338c40" containerName="maas-api" Jul 18 03:09:57.406468 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:09:57.406423 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1d958dfb-95d984596-sh7xb" Jul 18 03:09:57.409422 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:09:57.409381 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-1d958dfb-serving-cert\"" Jul 18 03:09:57.418328 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:09:57.418284 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1d958dfb-95d984596-sh7xb"] Jul 18 03:09:57.487197 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:09:57.487121 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe314044-ec90-4e8f-beaa-131526a4fa63-maas-api-tls\") pod \"maas-api-e2e-mt-1d958dfb-95d984596-sh7xb\" (UID: \"fe314044-ec90-4e8f-beaa-131526a4fa63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1d958dfb-95d984596-sh7xb" Jul 18 03:09:57.487197 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:09:57.487193 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe314044-ec90-4e8f-beaa-131526a4fa63-openshift-service-ca\") pod \"maas-api-e2e-mt-1d958dfb-95d984596-sh7xb\" (UID: \"fe314044-ec90-4e8f-beaa-131526a4fa63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1d958dfb-95d984596-sh7xb" Jul 18 03:09:57.487561 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:09:57.487224 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvjxf\" (UniqueName: \"kubernetes.io/projected/fe314044-ec90-4e8f-beaa-131526a4fa63-kube-api-access-rvjxf\") pod \"maas-api-e2e-mt-1d958dfb-95d984596-sh7xb\" (UID: \"fe314044-ec90-4e8f-beaa-131526a4fa63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1d958dfb-95d984596-sh7xb" Jul 18 03:09:57.588170 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:09:57.588100 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe314044-ec90-4e8f-beaa-131526a4fa63-maas-api-tls\") pod \"maas-api-e2e-mt-1d958dfb-95d984596-sh7xb\" (UID: \"fe314044-ec90-4e8f-beaa-131526a4fa63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1d958dfb-95d984596-sh7xb" Jul 18 03:09:57.588170 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:09:57.588176 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe314044-ec90-4e8f-beaa-131526a4fa63-openshift-service-ca\") pod \"maas-api-e2e-mt-1d958dfb-95d984596-sh7xb\" (UID: \"fe314044-ec90-4e8f-beaa-131526a4fa63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1d958dfb-95d984596-sh7xb" Jul 18 03:09:57.588549 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:09:57.588228 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvjxf\" (UniqueName: \"kubernetes.io/projected/fe314044-ec90-4e8f-beaa-131526a4fa63-kube-api-access-rvjxf\") pod \"maas-api-e2e-mt-1d958dfb-95d984596-sh7xb\" (UID: \"fe314044-ec90-4e8f-beaa-131526a4fa63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1d958dfb-95d984596-sh7xb" Jul 18 03:09:57.589093 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:09:57.589057 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe314044-ec90-4e8f-beaa-131526a4fa63-openshift-service-ca\") pod \"maas-api-e2e-mt-1d958dfb-95d984596-sh7xb\" (UID: \"fe314044-ec90-4e8f-beaa-131526a4fa63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1d958dfb-95d984596-sh7xb" Jul 18 03:09:57.591716 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:09:57.591675 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe314044-ec90-4e8f-beaa-131526a4fa63-maas-api-tls\") pod \"maas-api-e2e-mt-1d958dfb-95d984596-sh7xb\" (UID: \"fe314044-ec90-4e8f-beaa-131526a4fa63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1d958dfb-95d984596-sh7xb" Jul 18 03:09:57.598494 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:09:57.598448 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvjxf\" (UniqueName: \"kubernetes.io/projected/fe314044-ec90-4e8f-beaa-131526a4fa63-kube-api-access-rvjxf\") pod \"maas-api-e2e-mt-1d958dfb-95d984596-sh7xb\" (UID: \"fe314044-ec90-4e8f-beaa-131526a4fa63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1d958dfb-95d984596-sh7xb" Jul 18 03:09:57.722548 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:09:57.722474 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1d958dfb-95d984596-sh7xb" Jul 18 03:09:57.920493 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:09:57.920438 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1d958dfb-95d984596-sh7xb"] Jul 18 03:09:57.924508 ip-10-0-140-4 kubenswrapper[2585]: W0718 03:09:57.924455 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe314044_ec90_4e8f_beaa_131526a4fa63.slice/crio-6cb99c154c769822fd5fccf719a0c600f717d7c8137d0235bf547e6a5cd60dd7 WatchSource:0}: Error finding container 6cb99c154c769822fd5fccf719a0c600f717d7c8137d0235bf547e6a5cd60dd7: Status 404 returned error can't find the container with id 6cb99c154c769822fd5fccf719a0c600f717d7c8137d0235bf547e6a5cd60dd7 Jul 18 03:09:58.485480 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:09:58.485425 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1d958dfb-95d984596-sh7xb" event={"ID":"fe314044-ec90-4e8f-beaa-131526a4fa63","Type":"ContainerStarted","Data":"6cb99c154c769822fd5fccf719a0c600f717d7c8137d0235bf547e6a5cd60dd7"} Jul 18 03:09:59.493769 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:09:59.493706 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1d958dfb-95d984596-sh7xb" event={"ID":"fe314044-ec90-4e8f-beaa-131526a4fa63","Type":"ContainerStarted","Data":"c7e62417df8df3d2833a93d1e8e3b256aa993ff21ee6ca5cdee34adfb9879023"} Jul 18 03:09:59.494443 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:09:59.493811 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1d958dfb-95d984596-sh7xb" Jul 18 03:09:59.512674 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:09:59.512576 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1d958dfb-95d984596-sh7xb" podStartSLOduration=1.9776293649999999 podStartE2EDuration="2.512549585s" totalImagesPullingTime="534.92022ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 03:09:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 03:09:59.510256501 +0000 UTC m=+1823.113904189" watchObservedRunningTime="2026-07-18 03:09:59.512549585 +0000 UTC m=+1823.116197272" Jul 18 03:10:05.507480 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:05.507436 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1d958dfb-95d984596-sh7xb" Jul 18 03:10:19.195978 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:19.195936 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1d958dfb-95d984596-sh7xb"] Jul 18 03:10:19.196512 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:19.196227 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1d958dfb-95d984596-sh7xb" podUID="fe314044-ec90-4e8f-beaa-131526a4fa63" containerName="maas-api" containerID="cri-o://c7e62417df8df3d2833a93d1e8e3b256aa993ff21ee6ca5cdee34adfb9879023" gracePeriod=30 Jul 18 03:10:19.466454 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:19.466422 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1d958dfb-95d984596-sh7xb" Jul 18 03:10:19.603156 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:19.603113 2585 generic.go:356] "Generic (PLEG): container finished" podID="fe314044-ec90-4e8f-beaa-131526a4fa63" containerID="c7e62417df8df3d2833a93d1e8e3b256aa993ff21ee6ca5cdee34adfb9879023" exitCode=0 Jul 18 03:10:19.603407 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:19.603164 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1d958dfb-95d984596-sh7xb" Jul 18 03:10:19.603407 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:19.603213 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1d958dfb-95d984596-sh7xb" event={"ID":"fe314044-ec90-4e8f-beaa-131526a4fa63","Type":"ContainerDied","Data":"c7e62417df8df3d2833a93d1e8e3b256aa993ff21ee6ca5cdee34adfb9879023"} Jul 18 03:10:19.603407 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:19.603267 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1d958dfb-95d984596-sh7xb" event={"ID":"fe314044-ec90-4e8f-beaa-131526a4fa63","Type":"ContainerDied","Data":"6cb99c154c769822fd5fccf719a0c600f717d7c8137d0235bf547e6a5cd60dd7"} Jul 18 03:10:19.603407 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:19.603292 2585 scope.go:117] "RemoveContainer" containerID="c7e62417df8df3d2833a93d1e8e3b256aa993ff21ee6ca5cdee34adfb9879023" Jul 18 03:10:19.612468 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:19.612445 2585 scope.go:117] "RemoveContainer" containerID="c7e62417df8df3d2833a93d1e8e3b256aa993ff21ee6ca5cdee34adfb9879023" Jul 18 03:10:19.612819 ip-10-0-140-4 kubenswrapper[2585]: E0718 03:10:19.612800 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7e62417df8df3d2833a93d1e8e3b256aa993ff21ee6ca5cdee34adfb9879023\": container with ID starting with c7e62417df8df3d2833a93d1e8e3b256aa993ff21ee6ca5cdee34adfb9879023 not found: ID does not exist" containerID="c7e62417df8df3d2833a93d1e8e3b256aa993ff21ee6ca5cdee34adfb9879023" Jul 18 03:10:19.612860 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:19.612831 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7e62417df8df3d2833a93d1e8e3b256aa993ff21ee6ca5cdee34adfb9879023"} err="failed to get container status \"c7e62417df8df3d2833a93d1e8e3b256aa993ff21ee6ca5cdee34adfb9879023\": rpc error: code = NotFound desc = could not find container \"c7e62417df8df3d2833a93d1e8e3b256aa993ff21ee6ca5cdee34adfb9879023\": container with ID starting with c7e62417df8df3d2833a93d1e8e3b256aa993ff21ee6ca5cdee34adfb9879023 not found: ID does not exist" Jul 18 03:10:19.617148 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:19.617122 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe314044-ec90-4e8f-beaa-131526a4fa63-maas-api-tls\") pod \"fe314044-ec90-4e8f-beaa-131526a4fa63\" (UID: \"fe314044-ec90-4e8f-beaa-131526a4fa63\") " Jul 18 03:10:19.617282 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:19.617241 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe314044-ec90-4e8f-beaa-131526a4fa63-openshift-service-ca\") pod \"fe314044-ec90-4e8f-beaa-131526a4fa63\" (UID: \"fe314044-ec90-4e8f-beaa-131526a4fa63\") " Jul 18 03:10:19.617282 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:19.617266 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvjxf\" (UniqueName: \"kubernetes.io/projected/fe314044-ec90-4e8f-beaa-131526a4fa63-kube-api-access-rvjxf\") pod \"fe314044-ec90-4e8f-beaa-131526a4fa63\" (UID: \"fe314044-ec90-4e8f-beaa-131526a4fa63\") " Jul 18 03:10:19.617588 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:19.617564 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe314044-ec90-4e8f-beaa-131526a4fa63-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fe314044-ec90-4e8f-beaa-131526a4fa63" (UID: "fe314044-ec90-4e8f-beaa-131526a4fa63"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 03:10:19.619379 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:19.619357 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe314044-ec90-4e8f-beaa-131526a4fa63-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fe314044-ec90-4e8f-beaa-131526a4fa63" (UID: "fe314044-ec90-4e8f-beaa-131526a4fa63"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 03:10:19.619464 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:19.619359 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe314044-ec90-4e8f-beaa-131526a4fa63-kube-api-access-rvjxf" (OuterVolumeSpecName: "kube-api-access-rvjxf") pod "fe314044-ec90-4e8f-beaa-131526a4fa63" (UID: "fe314044-ec90-4e8f-beaa-131526a4fa63"). InnerVolumeSpecName "kube-api-access-rvjxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 03:10:19.718922 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:19.718886 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe314044-ec90-4e8f-beaa-131526a4fa63-openshift-service-ca\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:10:19.719078 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:19.718951 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvjxf\" (UniqueName: \"kubernetes.io/projected/fe314044-ec90-4e8f-beaa-131526a4fa63-kube-api-access-rvjxf\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:10:19.719078 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:19.718963 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe314044-ec90-4e8f-beaa-131526a4fa63-maas-api-tls\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:10:19.925238 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:19.925192 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1d958dfb-95d984596-sh7xb"] Jul 18 03:10:19.927000 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:19.926980 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1d958dfb-95d984596-sh7xb"] Jul 18 03:10:20.901153 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:20.901107 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe314044-ec90-4e8f-beaa-131526a4fa63" path="/var/lib/kubelet/pods/fe314044-ec90-4e8f-beaa-131526a4fa63/volumes" Jul 18 03:10:52.590707 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:52.590672 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk"] Jul 18 03:10:52.591284 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:52.591264 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe314044-ec90-4e8f-beaa-131526a4fa63" containerName="maas-api" Jul 18 03:10:52.591337 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:52.591288 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe314044-ec90-4e8f-beaa-131526a4fa63" containerName="maas-api" Jul 18 03:10:52.591419 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:52.591407 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe314044-ec90-4e8f-beaa-131526a4fa63" containerName="maas-api" Jul 18 03:10:52.594846 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:52.594826 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk" Jul 18 03:10:52.598226 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:52.598203 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7aa900e3-serving-cert\"" Jul 18 03:10:52.604251 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:52.604226 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk"] Jul 18 03:10:52.693130 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:52.693096 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45rx7\" (UniqueName: \"kubernetes.io/projected/a6bf09e1-6601-4434-9853-0f88f639adf3-kube-api-access-45rx7\") pod \"maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk\" (UID: \"a6bf09e1-6601-4434-9853-0f88f639adf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk" Jul 18 03:10:52.693345 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:52.693141 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6bf09e1-6601-4434-9853-0f88f639adf3-openshift-service-ca\") pod \"maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk\" (UID: \"a6bf09e1-6601-4434-9853-0f88f639adf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk" Jul 18 03:10:52.693345 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:52.693211 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6bf09e1-6601-4434-9853-0f88f639adf3-maas-api-tls\") pod \"maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk\" (UID: \"a6bf09e1-6601-4434-9853-0f88f639adf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk" Jul 18 03:10:52.794151 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:52.794118 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45rx7\" (UniqueName: \"kubernetes.io/projected/a6bf09e1-6601-4434-9853-0f88f639adf3-kube-api-access-45rx7\") pod \"maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk\" (UID: \"a6bf09e1-6601-4434-9853-0f88f639adf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk" Jul 18 03:10:52.794320 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:52.794170 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6bf09e1-6601-4434-9853-0f88f639adf3-openshift-service-ca\") pod \"maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk\" (UID: \"a6bf09e1-6601-4434-9853-0f88f639adf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk" Jul 18 03:10:52.794320 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:52.794236 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6bf09e1-6601-4434-9853-0f88f639adf3-maas-api-tls\") pod \"maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk\" (UID: \"a6bf09e1-6601-4434-9853-0f88f639adf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk" Jul 18 03:10:52.794875 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:52.794850 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6bf09e1-6601-4434-9853-0f88f639adf3-openshift-service-ca\") pod \"maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk\" (UID: \"a6bf09e1-6601-4434-9853-0f88f639adf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk" Jul 18 03:10:52.796515 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:52.796484 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6bf09e1-6601-4434-9853-0f88f639adf3-maas-api-tls\") pod \"maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk\" (UID: \"a6bf09e1-6601-4434-9853-0f88f639adf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk" Jul 18 03:10:52.803788 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:52.803765 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45rx7\" (UniqueName: \"kubernetes.io/projected/a6bf09e1-6601-4434-9853-0f88f639adf3-kube-api-access-45rx7\") pod \"maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk\" (UID: \"a6bf09e1-6601-4434-9853-0f88f639adf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk" Jul 18 03:10:52.906223 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:52.906109 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk" Jul 18 03:10:53.244908 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:53.244881 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk"] Jul 18 03:10:53.246845 ip-10-0-140-4 kubenswrapper[2585]: W0718 03:10:53.246818 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6bf09e1_6601_4434_9853_0f88f639adf3.slice/crio-fd6ae5b1fc4ea950c75f040fb20c5a3c245bbcfb7193ef37cda08579575f2d39 WatchSource:0}: Error finding container fd6ae5b1fc4ea950c75f040fb20c5a3c245bbcfb7193ef37cda08579575f2d39: Status 404 returned error can't find the container with id fd6ae5b1fc4ea950c75f040fb20c5a3c245bbcfb7193ef37cda08579575f2d39 Jul 18 03:10:53.248045 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:53.248026 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 03:10:53.762558 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:53.762521 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk" event={"ID":"a6bf09e1-6601-4434-9853-0f88f639adf3","Type":"ContainerStarted","Data":"fd6ae5b1fc4ea950c75f040fb20c5a3c245bbcfb7193ef37cda08579575f2d39"} Jul 18 03:10:54.769199 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:54.769151 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk" event={"ID":"a6bf09e1-6601-4434-9853-0f88f639adf3","Type":"ContainerStarted","Data":"156513c30459114567d5d4fe15e8d7a856f31f43a43b39904c4bbe58f218d0a0"} Jul 18 03:10:54.769565 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:54.769278 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk" Jul 18 03:10:54.785730 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:10:54.785688 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk" podStartSLOduration=2.177344461 podStartE2EDuration="2.785673936s" totalImagesPullingTime="608.329475ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 03:10:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 03:10:54.784092803 +0000 UTC m=+1878.387740491" watchObservedRunningTime="2026-07-18 03:10:54.785673936 +0000 UTC m=+1878.389321623" Jul 18 03:11:00.780776 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:11:00.780748 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk" Jul 18 03:11:50.976135 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:11:50.976101 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk"] Jul 18 03:11:50.978785 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:11:50.976313 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk" podUID="a6bf09e1-6601-4434-9853-0f88f639adf3" containerName="maas-api" containerID="cri-o://156513c30459114567d5d4fe15e8d7a856f31f43a43b39904c4bbe58f218d0a0" gracePeriod=30 Jul 18 03:11:51.233918 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:11:51.233858 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk" Jul 18 03:11:51.326417 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:11:51.326385 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6bf09e1-6601-4434-9853-0f88f639adf3-maas-api-tls\") pod \"a6bf09e1-6601-4434-9853-0f88f639adf3\" (UID: \"a6bf09e1-6601-4434-9853-0f88f639adf3\") " Jul 18 03:11:51.326599 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:11:51.326434 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45rx7\" (UniqueName: \"kubernetes.io/projected/a6bf09e1-6601-4434-9853-0f88f639adf3-kube-api-access-45rx7\") pod \"a6bf09e1-6601-4434-9853-0f88f639adf3\" (UID: \"a6bf09e1-6601-4434-9853-0f88f639adf3\") " Jul 18 03:11:51.326599 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:11:51.326527 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6bf09e1-6601-4434-9853-0f88f639adf3-openshift-service-ca\") pod \"a6bf09e1-6601-4434-9853-0f88f639adf3\" (UID: \"a6bf09e1-6601-4434-9853-0f88f639adf3\") " Jul 18 03:11:51.326888 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:11:51.326862 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6bf09e1-6601-4434-9853-0f88f639adf3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a6bf09e1-6601-4434-9853-0f88f639adf3" (UID: "a6bf09e1-6601-4434-9853-0f88f639adf3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 03:11:51.328568 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:11:51.328537 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6bf09e1-6601-4434-9853-0f88f639adf3-kube-api-access-45rx7" (OuterVolumeSpecName: "kube-api-access-45rx7") pod "a6bf09e1-6601-4434-9853-0f88f639adf3" (UID: "a6bf09e1-6601-4434-9853-0f88f639adf3"). InnerVolumeSpecName "kube-api-access-45rx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 03:11:51.328686 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:11:51.328566 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6bf09e1-6601-4434-9853-0f88f639adf3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a6bf09e1-6601-4434-9853-0f88f639adf3" (UID: "a6bf09e1-6601-4434-9853-0f88f639adf3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 03:11:51.427912 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:11:51.427872 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6bf09e1-6601-4434-9853-0f88f639adf3-openshift-service-ca\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:11:51.427912 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:11:51.427901 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6bf09e1-6601-4434-9853-0f88f639adf3-maas-api-tls\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:11:51.427912 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:11:51.427910 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45rx7\" (UniqueName: \"kubernetes.io/projected/a6bf09e1-6601-4434-9853-0f88f639adf3-kube-api-access-45rx7\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:11:52.055409 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:11:52.055370 2585 generic.go:356] "Generic (PLEG): container finished" podID="a6bf09e1-6601-4434-9853-0f88f639adf3" containerID="156513c30459114567d5d4fe15e8d7a856f31f43a43b39904c4bbe58f218d0a0" exitCode=0 Jul 18 03:11:52.055409 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:11:52.055410 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk" Jul 18 03:11:52.055937 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:11:52.055454 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk" event={"ID":"a6bf09e1-6601-4434-9853-0f88f639adf3","Type":"ContainerDied","Data":"156513c30459114567d5d4fe15e8d7a856f31f43a43b39904c4bbe58f218d0a0"} Jul 18 03:11:52.055937 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:11:52.055506 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk" event={"ID":"a6bf09e1-6601-4434-9853-0f88f639adf3","Type":"ContainerDied","Data":"fd6ae5b1fc4ea950c75f040fb20c5a3c245bbcfb7193ef37cda08579575f2d39"} Jul 18 03:11:52.055937 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:11:52.055530 2585 scope.go:117] "RemoveContainer" containerID="156513c30459114567d5d4fe15e8d7a856f31f43a43b39904c4bbe58f218d0a0" Jul 18 03:11:52.065271 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:11:52.065256 2585 scope.go:117] "RemoveContainer" containerID="156513c30459114567d5d4fe15e8d7a856f31f43a43b39904c4bbe58f218d0a0" Jul 18 03:11:52.065536 ip-10-0-140-4 kubenswrapper[2585]: E0718 03:11:52.065514 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"156513c30459114567d5d4fe15e8d7a856f31f43a43b39904c4bbe58f218d0a0\": container with ID starting with 156513c30459114567d5d4fe15e8d7a856f31f43a43b39904c4bbe58f218d0a0 not found: ID does not exist" containerID="156513c30459114567d5d4fe15e8d7a856f31f43a43b39904c4bbe58f218d0a0" Jul 18 03:11:52.065637 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:11:52.065541 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"156513c30459114567d5d4fe15e8d7a856f31f43a43b39904c4bbe58f218d0a0"} err="failed to get container status \"156513c30459114567d5d4fe15e8d7a856f31f43a43b39904c4bbe58f218d0a0\": rpc error: code = NotFound desc = could not find container \"156513c30459114567d5d4fe15e8d7a856f31f43a43b39904c4bbe58f218d0a0\": container with ID starting with 156513c30459114567d5d4fe15e8d7a856f31f43a43b39904c4bbe58f218d0a0 not found: ID does not exist" Jul 18 03:11:52.076470 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:11:52.076448 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk"] Jul 18 03:11:52.079801 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:11:52.079781 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7aa900e3-794d55b649-5hfwk"] Jul 18 03:11:52.900924 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:11:52.900881 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6bf09e1-6601-4434-9853-0f88f639adf3" path="/var/lib/kubelet/pods/a6bf09e1-6601-4434-9853-0f88f639adf3/volumes" Jul 18 03:13:44.644203 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:44.644152 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m"] Jul 18 03:13:44.644763 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:44.644745 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6bf09e1-6601-4434-9853-0f88f639adf3" containerName="maas-api" Jul 18 03:13:44.644810 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:44.644766 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6bf09e1-6601-4434-9853-0f88f639adf3" containerName="maas-api" Jul 18 03:13:44.644877 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:44.644866 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6bf09e1-6601-4434-9853-0f88f639adf3" containerName="maas-api" Jul 18 03:13:44.653258 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:44.653233 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m" Jul 18 03:13:44.656005 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:44.655984 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-69f74a-serving-cert\"" Jul 18 03:13:44.656400 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:44.656372 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m"] Jul 18 03:13:44.802285 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:44.802248 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e18d770-dc33-41ef-8781-7de7f038d0a5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m\" (UID: \"4e18d770-dc33-41ef-8781-7de7f038d0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m" Jul 18 03:13:44.802466 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:44.802293 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e18d770-dc33-41ef-8781-7de7f038d0a5-maas-api-tls\") pod \"maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m\" (UID: \"4e18d770-dc33-41ef-8781-7de7f038d0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m" Jul 18 03:13:44.802466 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:44.802341 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qjzw\" (UniqueName: \"kubernetes.io/projected/4e18d770-dc33-41ef-8781-7de7f038d0a5-kube-api-access-5qjzw\") pod \"maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m\" (UID: \"4e18d770-dc33-41ef-8781-7de7f038d0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m" Jul 18 03:13:44.902819 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:44.902724 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e18d770-dc33-41ef-8781-7de7f038d0a5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m\" (UID: \"4e18d770-dc33-41ef-8781-7de7f038d0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m" Jul 18 03:13:44.902819 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:44.902777 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e18d770-dc33-41ef-8781-7de7f038d0a5-maas-api-tls\") pod \"maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m\" (UID: \"4e18d770-dc33-41ef-8781-7de7f038d0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m" Jul 18 03:13:44.902819 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:44.902804 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qjzw\" (UniqueName: \"kubernetes.io/projected/4e18d770-dc33-41ef-8781-7de7f038d0a5-kube-api-access-5qjzw\") pod \"maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m\" (UID: \"4e18d770-dc33-41ef-8781-7de7f038d0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m" Jul 18 03:13:44.903372 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:44.903344 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e18d770-dc33-41ef-8781-7de7f038d0a5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m\" (UID: \"4e18d770-dc33-41ef-8781-7de7f038d0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m" Jul 18 03:13:44.905353 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:44.905329 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e18d770-dc33-41ef-8781-7de7f038d0a5-maas-api-tls\") pod \"maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m\" (UID: \"4e18d770-dc33-41ef-8781-7de7f038d0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m" Jul 18 03:13:44.911350 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:44.911327 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qjzw\" (UniqueName: \"kubernetes.io/projected/4e18d770-dc33-41ef-8781-7de7f038d0a5-kube-api-access-5qjzw\") pod \"maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m\" (UID: \"4e18d770-dc33-41ef-8781-7de7f038d0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m" Jul 18 03:13:44.964387 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:44.964353 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m" Jul 18 03:13:45.102137 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:45.102109 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m"] Jul 18 03:13:45.104204 ip-10-0-140-4 kubenswrapper[2585]: W0718 03:13:45.104159 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e18d770_dc33_41ef_8781_7de7f038d0a5.slice/crio-d84c957c8a897672674c6e0a69a8c3ca0e6f3fb4c002b1f64987ecfb4bcb0bad WatchSource:0}: Error finding container d84c957c8a897672674c6e0a69a8c3ca0e6f3fb4c002b1f64987ecfb4bcb0bad: Status 404 returned error can't find the container with id d84c957c8a897672674c6e0a69a8c3ca0e6f3fb4c002b1f64987ecfb4bcb0bad Jul 18 03:13:45.588499 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:45.588466 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m" event={"ID":"4e18d770-dc33-41ef-8781-7de7f038d0a5","Type":"ContainerStarted","Data":"d84c957c8a897672674c6e0a69a8c3ca0e6f3fb4c002b1f64987ecfb4bcb0bad"} Jul 18 03:13:46.595214 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:46.595170 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m" event={"ID":"4e18d770-dc33-41ef-8781-7de7f038d0a5","Type":"ContainerStarted","Data":"b967d8b1e08aaa173596f574341b3a693725c5d8516f466706c4dead19de5dc1"} Jul 18 03:13:46.595581 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:46.595249 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m" Jul 18 03:13:46.612229 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:46.612165 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m" podStartSLOduration=2.038076018 podStartE2EDuration="2.612150915s" totalImagesPullingTime="574.074897ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 03:13:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 03:13:46.610674785 +0000 UTC m=+2050.214322473" watchObservedRunningTime="2026-07-18 03:13:46.612150915 +0000 UTC m=+2050.215798602" Jul 18 03:13:52.580457 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:52.580415 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw"] Jul 18 03:13:52.584856 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:52.584835 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw" Jul 18 03:13:52.591699 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:52.591677 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-cde1c8-serving-cert\"" Jul 18 03:13:52.598605 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:52.598577 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw"] Jul 18 03:13:52.605652 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:52.605628 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m" Jul 18 03:13:52.772174 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:52.772141 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0af40f92-a8fc-4b68-8fff-99da577ca256-maas-api-tls\") pod \"maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw\" (UID: \"0af40f92-a8fc-4b68-8fff-99da577ca256\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw" Jul 18 03:13:52.772361 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:52.772228 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwjtb\" (UniqueName: \"kubernetes.io/projected/0af40f92-a8fc-4b68-8fff-99da577ca256-kube-api-access-vwjtb\") pod \"maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw\" (UID: \"0af40f92-a8fc-4b68-8fff-99da577ca256\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw" Jul 18 03:13:52.772437 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:52.772377 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af40f92-a8fc-4b68-8fff-99da577ca256-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw\" (UID: \"0af40f92-a8fc-4b68-8fff-99da577ca256\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw" Jul 18 03:13:52.873913 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:52.873804 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwjtb\" (UniqueName: \"kubernetes.io/projected/0af40f92-a8fc-4b68-8fff-99da577ca256-kube-api-access-vwjtb\") pod \"maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw\" (UID: \"0af40f92-a8fc-4b68-8fff-99da577ca256\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw" Jul 18 03:13:52.873913 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:52.873888 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af40f92-a8fc-4b68-8fff-99da577ca256-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw\" (UID: \"0af40f92-a8fc-4b68-8fff-99da577ca256\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw" Jul 18 03:13:52.874148 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:52.873930 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0af40f92-a8fc-4b68-8fff-99da577ca256-maas-api-tls\") pod \"maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw\" (UID: \"0af40f92-a8fc-4b68-8fff-99da577ca256\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw" Jul 18 03:13:52.874579 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:52.874551 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af40f92-a8fc-4b68-8fff-99da577ca256-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw\" (UID: \"0af40f92-a8fc-4b68-8fff-99da577ca256\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw" Jul 18 03:13:52.876471 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:52.876449 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0af40f92-a8fc-4b68-8fff-99da577ca256-maas-api-tls\") pod \"maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw\" (UID: \"0af40f92-a8fc-4b68-8fff-99da577ca256\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw" Jul 18 03:13:52.885591 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:52.885567 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwjtb\" (UniqueName: \"kubernetes.io/projected/0af40f92-a8fc-4b68-8fff-99da577ca256-kube-api-access-vwjtb\") pod \"maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw\" (UID: \"0af40f92-a8fc-4b68-8fff-99da577ca256\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw" Jul 18 03:13:52.894478 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:52.894460 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw" Jul 18 03:13:53.034537 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:53.034510 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw"] Jul 18 03:13:53.036645 ip-10-0-140-4 kubenswrapper[2585]: W0718 03:13:53.036618 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af40f92_a8fc_4b68_8fff_99da577ca256.slice/crio-28c1322524db4d369ef385316642efd722dba54c86221cd3cc8dafcd29cfa4d6 WatchSource:0}: Error finding container 28c1322524db4d369ef385316642efd722dba54c86221cd3cc8dafcd29cfa4d6: Status 404 returned error can't find the container with id 28c1322524db4d369ef385316642efd722dba54c86221cd3cc8dafcd29cfa4d6 Jul 18 03:13:53.630239 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:53.630175 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw" event={"ID":"0af40f92-a8fc-4b68-8fff-99da577ca256","Type":"ContainerStarted","Data":"864db5bdadad50a7008e14956b230e494731582116c5b3e21b4895235bd1fb21"} Jul 18 03:13:53.630765 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:53.630256 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw" event={"ID":"0af40f92-a8fc-4b68-8fff-99da577ca256","Type":"ContainerStarted","Data":"28c1322524db4d369ef385316642efd722dba54c86221cd3cc8dafcd29cfa4d6"} Jul 18 03:13:53.630765 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:53.630406 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw" Jul 18 03:13:53.649690 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:53.649587 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw" podStartSLOduration=1.303318872 podStartE2EDuration="1.649571302s" totalImagesPullingTime="346.25243ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 03:13:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 03:13:53.647719873 +0000 UTC m=+2057.251367560" watchObservedRunningTime="2026-07-18 03:13:53.649571302 +0000 UTC m=+2057.253219046" Jul 18 03:13:59.641106 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:13:59.641024 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw" Jul 18 03:14:37.032577 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:14:37.032545 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-krtsg_709575dc-6eae-43e1-9208-cc228795929a/console-operator/1.log" Jul 18 03:14:37.041049 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:14:37.041026 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-krtsg_709575dc-6eae-43e1-9208-cc228795929a/console-operator/1.log" Jul 18 03:14:51.706766 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:14:51.706729 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m"] Jul 18 03:14:51.707289 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:14:51.706987 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m" podUID="4e18d770-dc33-41ef-8781-7de7f038d0a5" containerName="maas-api" containerID="cri-o://b967d8b1e08aaa173596f574341b3a693725c5d8516f466706c4dead19de5dc1" gracePeriod=30 Jul 18 03:14:51.928871 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:14:51.928790 2585 generic.go:356] "Generic (PLEG): container finished" podID="4e18d770-dc33-41ef-8781-7de7f038d0a5" containerID="b967d8b1e08aaa173596f574341b3a693725c5d8516f466706c4dead19de5dc1" exitCode=0 Jul 18 03:14:51.928871 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:14:51.928839 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m" event={"ID":"4e18d770-dc33-41ef-8781-7de7f038d0a5","Type":"ContainerDied","Data":"b967d8b1e08aaa173596f574341b3a693725c5d8516f466706c4dead19de5dc1"} Jul 18 03:14:51.958267 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:14:51.958209 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m" Jul 18 03:14:51.992131 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:14:51.992099 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e18d770-dc33-41ef-8781-7de7f038d0a5-maas-api-tls\") pod \"4e18d770-dc33-41ef-8781-7de7f038d0a5\" (UID: \"4e18d770-dc33-41ef-8781-7de7f038d0a5\") " Jul 18 03:14:51.992325 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:14:51.992250 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qjzw\" (UniqueName: \"kubernetes.io/projected/4e18d770-dc33-41ef-8781-7de7f038d0a5-kube-api-access-5qjzw\") pod \"4e18d770-dc33-41ef-8781-7de7f038d0a5\" (UID: \"4e18d770-dc33-41ef-8781-7de7f038d0a5\") " Jul 18 03:14:51.992325 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:14:51.992291 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e18d770-dc33-41ef-8781-7de7f038d0a5-openshift-service-ca\") pod \"4e18d770-dc33-41ef-8781-7de7f038d0a5\" (UID: \"4e18d770-dc33-41ef-8781-7de7f038d0a5\") " Jul 18 03:14:51.992658 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:14:51.992630 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e18d770-dc33-41ef-8781-7de7f038d0a5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4e18d770-dc33-41ef-8781-7de7f038d0a5" (UID: "4e18d770-dc33-41ef-8781-7de7f038d0a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 03:14:51.994312 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:14:51.994291 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e18d770-dc33-41ef-8781-7de7f038d0a5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4e18d770-dc33-41ef-8781-7de7f038d0a5" (UID: "4e18d770-dc33-41ef-8781-7de7f038d0a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 03:14:51.994442 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:14:51.994421 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e18d770-dc33-41ef-8781-7de7f038d0a5-kube-api-access-5qjzw" (OuterVolumeSpecName: "kube-api-access-5qjzw") pod "4e18d770-dc33-41ef-8781-7de7f038d0a5" (UID: "4e18d770-dc33-41ef-8781-7de7f038d0a5"). InnerVolumeSpecName "kube-api-access-5qjzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 03:14:52.093155 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:14:52.093115 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qjzw\" (UniqueName: \"kubernetes.io/projected/4e18d770-dc33-41ef-8781-7de7f038d0a5-kube-api-access-5qjzw\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:14:52.093155 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:14:52.093149 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e18d770-dc33-41ef-8781-7de7f038d0a5-openshift-service-ca\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:14:52.093155 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:14:52.093159 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e18d770-dc33-41ef-8781-7de7f038d0a5-maas-api-tls\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:14:52.935513 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:14:52.935479 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m" Jul 18 03:14:52.935513 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:14:52.935499 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m" event={"ID":"4e18d770-dc33-41ef-8781-7de7f038d0a5","Type":"ContainerDied","Data":"d84c957c8a897672674c6e0a69a8c3ca0e6f3fb4c002b1f64987ecfb4bcb0bad"} Jul 18 03:14:52.935983 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:14:52.935566 2585 scope.go:117] "RemoveContainer" containerID="b967d8b1e08aaa173596f574341b3a693725c5d8516f466706c4dead19de5dc1" Jul 18 03:14:52.952166 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:14:52.952138 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m"] Jul 18 03:14:52.953769 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:14:52.953748 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-69f74a-b9657f5d4-8kb2m"] Jul 18 03:14:54.900746 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:14:54.900702 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e18d770-dc33-41ef-8781-7de7f038d0a5" path="/var/lib/kubelet/pods/4e18d770-dc33-41ef-8781-7de7f038d0a5/volumes" Jul 18 03:15:14.143506 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:14.143454 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw"] Jul 18 03:15:14.144274 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:14.143696 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw" podUID="0af40f92-a8fc-4b68-8fff-99da577ca256" containerName="maas-api" containerID="cri-o://864db5bdadad50a7008e14956b230e494731582116c5b3e21b4895235bd1fb21" gracePeriod=30 Jul 18 03:15:14.416630 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:14.416588 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw" Jul 18 03:15:14.500546 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:14.500505 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwjtb\" (UniqueName: \"kubernetes.io/projected/0af40f92-a8fc-4b68-8fff-99da577ca256-kube-api-access-vwjtb\") pod \"0af40f92-a8fc-4b68-8fff-99da577ca256\" (UID: \"0af40f92-a8fc-4b68-8fff-99da577ca256\") " Jul 18 03:15:14.500781 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:14.500565 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0af40f92-a8fc-4b68-8fff-99da577ca256-maas-api-tls\") pod \"0af40f92-a8fc-4b68-8fff-99da577ca256\" (UID: \"0af40f92-a8fc-4b68-8fff-99da577ca256\") " Jul 18 03:15:14.500781 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:14.500665 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af40f92-a8fc-4b68-8fff-99da577ca256-openshift-service-ca\") pod \"0af40f92-a8fc-4b68-8fff-99da577ca256\" (UID: \"0af40f92-a8fc-4b68-8fff-99da577ca256\") " Jul 18 03:15:14.501034 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:14.501002 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af40f92-a8fc-4b68-8fff-99da577ca256-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0af40f92-a8fc-4b68-8fff-99da577ca256" (UID: "0af40f92-a8fc-4b68-8fff-99da577ca256"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 03:15:14.502952 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:14.502917 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0af40f92-a8fc-4b68-8fff-99da577ca256-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0af40f92-a8fc-4b68-8fff-99da577ca256" (UID: "0af40f92-a8fc-4b68-8fff-99da577ca256"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 03:15:14.503097 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:14.502948 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af40f92-a8fc-4b68-8fff-99da577ca256-kube-api-access-vwjtb" (OuterVolumeSpecName: "kube-api-access-vwjtb") pod "0af40f92-a8fc-4b68-8fff-99da577ca256" (UID: "0af40f92-a8fc-4b68-8fff-99da577ca256"). InnerVolumeSpecName "kube-api-access-vwjtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 03:15:14.601426 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:14.601375 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwjtb\" (UniqueName: \"kubernetes.io/projected/0af40f92-a8fc-4b68-8fff-99da577ca256-kube-api-access-vwjtb\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:15:14.601426 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:14.601415 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0af40f92-a8fc-4b68-8fff-99da577ca256-maas-api-tls\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:15:14.601426 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:14.601430 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af40f92-a8fc-4b68-8fff-99da577ca256-openshift-service-ca\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:15:15.056373 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:15.056330 2585 generic.go:356] "Generic (PLEG): container finished" podID="0af40f92-a8fc-4b68-8fff-99da577ca256" containerID="864db5bdadad50a7008e14956b230e494731582116c5b3e21b4895235bd1fb21" exitCode=0 Jul 18 03:15:15.056552 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:15.056395 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw" Jul 18 03:15:15.056552 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:15.056414 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw" event={"ID":"0af40f92-a8fc-4b68-8fff-99da577ca256","Type":"ContainerDied","Data":"864db5bdadad50a7008e14956b230e494731582116c5b3e21b4895235bd1fb21"} Jul 18 03:15:15.056552 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:15.056494 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw" event={"ID":"0af40f92-a8fc-4b68-8fff-99da577ca256","Type":"ContainerDied","Data":"28c1322524db4d369ef385316642efd722dba54c86221cd3cc8dafcd29cfa4d6"} Jul 18 03:15:15.056552 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:15.056517 2585 scope.go:117] "RemoveContainer" containerID="864db5bdadad50a7008e14956b230e494731582116c5b3e21b4895235bd1fb21" Jul 18 03:15:15.066346 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:15.066318 2585 scope.go:117] "RemoveContainer" containerID="864db5bdadad50a7008e14956b230e494731582116c5b3e21b4895235bd1fb21" Jul 18 03:15:15.066702 ip-10-0-140-4 kubenswrapper[2585]: E0718 03:15:15.066675 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"864db5bdadad50a7008e14956b230e494731582116c5b3e21b4895235bd1fb21\": container with ID starting with 864db5bdadad50a7008e14956b230e494731582116c5b3e21b4895235bd1fb21 not found: ID does not exist" containerID="864db5bdadad50a7008e14956b230e494731582116c5b3e21b4895235bd1fb21" Jul 18 03:15:15.066786 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:15.066715 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"864db5bdadad50a7008e14956b230e494731582116c5b3e21b4895235bd1fb21"} err="failed to get container status \"864db5bdadad50a7008e14956b230e494731582116c5b3e21b4895235bd1fb21\": rpc error: code = NotFound desc = could not find container \"864db5bdadad50a7008e14956b230e494731582116c5b3e21b4895235bd1fb21\": container with ID starting with 864db5bdadad50a7008e14956b230e494731582116c5b3e21b4895235bd1fb21 not found: ID does not exist" Jul 18 03:15:15.074908 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:15.074868 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw"] Jul 18 03:15:15.079373 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:15.079335 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cde1c8-78c855b484-ld8xw"] Jul 18 03:15:16.901456 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:16.901415 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0af40f92-a8fc-4b68-8fff-99da577ca256" path="/var/lib/kubelet/pods/0af40f92-a8fc-4b68-8fff-99da577ca256/volumes" Jul 18 03:15:29.188560 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:29.188514 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf"] Jul 18 03:15:29.189117 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:29.188943 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e18d770-dc33-41ef-8781-7de7f038d0a5" containerName="maas-api" Jul 18 03:15:29.189117 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:29.188956 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e18d770-dc33-41ef-8781-7de7f038d0a5" containerName="maas-api" Jul 18 03:15:29.189117 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:29.188972 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0af40f92-a8fc-4b68-8fff-99da577ca256" containerName="maas-api" Jul 18 03:15:29.189117 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:29.188979 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0af40f92-a8fc-4b68-8fff-99da577ca256" containerName="maas-api" Jul 18 03:15:29.189117 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:29.189056 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e18d770-dc33-41ef-8781-7de7f038d0a5" containerName="maas-api" Jul 18 03:15:29.189117 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:29.189068 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0af40f92-a8fc-4b68-8fff-99da577ca256" containerName="maas-api" Jul 18 03:15:29.193609 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:29.193571 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf" Jul 18 03:15:29.196914 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:29.196866 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-4e7186-serving-cert\"" Jul 18 03:15:29.206357 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:29.206324 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf"] Jul 18 03:15:29.241879 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:29.241776 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc9n5\" (UniqueName: \"kubernetes.io/projected/7ed6b041-48d6-43e7-82fb-c643cbf25755-kube-api-access-jc9n5\") pod \"maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf\" (UID: \"7ed6b041-48d6-43e7-82fb-c643cbf25755\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf" Jul 18 03:15:29.241879 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:29.241861 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ed6b041-48d6-43e7-82fb-c643cbf25755-maas-api-tls\") pod \"maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf\" (UID: \"7ed6b041-48d6-43e7-82fb-c643cbf25755\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf" Jul 18 03:15:29.242100 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:29.241938 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed6b041-48d6-43e7-82fb-c643cbf25755-openshift-service-ca\") pod \"maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf\" (UID: \"7ed6b041-48d6-43e7-82fb-c643cbf25755\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf" Jul 18 03:15:29.343263 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:29.343221 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ed6b041-48d6-43e7-82fb-c643cbf25755-maas-api-tls\") pod \"maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf\" (UID: \"7ed6b041-48d6-43e7-82fb-c643cbf25755\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf" Jul 18 03:15:29.343486 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:29.343335 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed6b041-48d6-43e7-82fb-c643cbf25755-openshift-service-ca\") pod \"maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf\" (UID: \"7ed6b041-48d6-43e7-82fb-c643cbf25755\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf" Jul 18 03:15:29.343486 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:29.343382 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc9n5\" (UniqueName: \"kubernetes.io/projected/7ed6b041-48d6-43e7-82fb-c643cbf25755-kube-api-access-jc9n5\") pod \"maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf\" (UID: \"7ed6b041-48d6-43e7-82fb-c643cbf25755\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf" Jul 18 03:15:29.343486 ip-10-0-140-4 kubenswrapper[2585]: E0718 03:15:29.343400 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-4e7186-serving-cert: secret "maas-api-e2e-api-a-4e7186-serving-cert" not found Jul 18 03:15:29.343486 ip-10-0-140-4 kubenswrapper[2585]: E0718 03:15:29.343483 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ed6b041-48d6-43e7-82fb-c643cbf25755-maas-api-tls podName:7ed6b041-48d6-43e7-82fb-c643cbf25755 nodeName:}" failed. No retries permitted until 2026-07-18 03:15:29.843458196 +0000 UTC m=+2153.447105880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7ed6b041-48d6-43e7-82fb-c643cbf25755-maas-api-tls") pod "maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf" (UID: "7ed6b041-48d6-43e7-82fb-c643cbf25755") : secret "maas-api-e2e-api-a-4e7186-serving-cert" not found Jul 18 03:15:29.344108 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:29.344084 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed6b041-48d6-43e7-82fb-c643cbf25755-openshift-service-ca\") pod \"maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf\" (UID: \"7ed6b041-48d6-43e7-82fb-c643cbf25755\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf" Jul 18 03:15:29.355941 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:29.355901 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc9n5\" (UniqueName: \"kubernetes.io/projected/7ed6b041-48d6-43e7-82fb-c643cbf25755-kube-api-access-jc9n5\") pod \"maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf\" (UID: \"7ed6b041-48d6-43e7-82fb-c643cbf25755\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf" Jul 18 03:15:29.847461 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:29.847408 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ed6b041-48d6-43e7-82fb-c643cbf25755-maas-api-tls\") pod \"maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf\" (UID: \"7ed6b041-48d6-43e7-82fb-c643cbf25755\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf" Jul 18 03:15:29.850290 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:29.850264 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ed6b041-48d6-43e7-82fb-c643cbf25755-maas-api-tls\") pod \"maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf\" (UID: \"7ed6b041-48d6-43e7-82fb-c643cbf25755\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf" Jul 18 03:15:30.106151 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:30.106050 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf" Jul 18 03:15:30.261001 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:30.260954 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf"] Jul 18 03:15:30.264899 ip-10-0-140-4 kubenswrapper[2585]: W0718 03:15:30.264862 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed6b041_48d6_43e7_82fb_c643cbf25755.slice/crio-4f74068dafd1f0f45727a11b994f694147ae99c49ab1fbcf6660379ebe922882 WatchSource:0}: Error finding container 4f74068dafd1f0f45727a11b994f694147ae99c49ab1fbcf6660379ebe922882: Status 404 returned error can't find the container with id 4f74068dafd1f0f45727a11b994f694147ae99c49ab1fbcf6660379ebe922882 Jul 18 03:15:31.144928 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:31.144817 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf" event={"ID":"7ed6b041-48d6-43e7-82fb-c643cbf25755","Type":"ContainerStarted","Data":"92512cff9136863824cb5ce8a1c167992d85e89f3a2cde7d57ab3b1c83995b27"} Jul 18 03:15:31.144928 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:31.144879 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf" event={"ID":"7ed6b041-48d6-43e7-82fb-c643cbf25755","Type":"ContainerStarted","Data":"4f74068dafd1f0f45727a11b994f694147ae99c49ab1fbcf6660379ebe922882"} Jul 18 03:15:31.144928 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:31.144906 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf" Jul 18 03:15:31.164103 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:31.164043 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf" podStartSLOduration=1.622484006 podStartE2EDuration="2.163998148s" totalImagesPullingTime="541.514142ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 03:15:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 03:15:31.163417813 +0000 UTC m=+2154.767065500" watchObservedRunningTime="2026-07-18 03:15:31.163998148 +0000 UTC m=+2154.767645835" Jul 18 03:15:37.156889 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:37.156854 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf" Jul 18 03:15:37.294682 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:37.294638 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k"] Jul 18 03:15:37.298712 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:37.298670 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k" Jul 18 03:15:37.301028 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:37.300996 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-bddfdd-serving-cert\"" Jul 18 03:15:37.309654 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:37.309617 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k"] Jul 18 03:15:37.321287 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:37.321252 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c6b1845-d3d3-4fc0-83e7-752707f373ae-maas-api-tls\") pod \"maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k\" (UID: \"1c6b1845-d3d3-4fc0-83e7-752707f373ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k" Jul 18 03:15:37.321540 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:37.321380 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rnmg\" (UniqueName: \"kubernetes.io/projected/1c6b1845-d3d3-4fc0-83e7-752707f373ae-kube-api-access-6rnmg\") pod \"maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k\" (UID: \"1c6b1845-d3d3-4fc0-83e7-752707f373ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k" Jul 18 03:15:37.321634 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:37.321541 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c6b1845-d3d3-4fc0-83e7-752707f373ae-openshift-service-ca\") pod \"maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k\" (UID: \"1c6b1845-d3d3-4fc0-83e7-752707f373ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k" Jul 18 03:15:37.423161 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:37.423057 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c6b1845-d3d3-4fc0-83e7-752707f373ae-openshift-service-ca\") pod \"maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k\" (UID: \"1c6b1845-d3d3-4fc0-83e7-752707f373ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k" Jul 18 03:15:37.423161 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:37.423126 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c6b1845-d3d3-4fc0-83e7-752707f373ae-maas-api-tls\") pod \"maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k\" (UID: \"1c6b1845-d3d3-4fc0-83e7-752707f373ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k" Jul 18 03:15:37.423161 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:37.423156 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rnmg\" (UniqueName: \"kubernetes.io/projected/1c6b1845-d3d3-4fc0-83e7-752707f373ae-kube-api-access-6rnmg\") pod \"maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k\" (UID: \"1c6b1845-d3d3-4fc0-83e7-752707f373ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k" Jul 18 03:15:37.423890 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:37.423859 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c6b1845-d3d3-4fc0-83e7-752707f373ae-openshift-service-ca\") pod \"maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k\" (UID: \"1c6b1845-d3d3-4fc0-83e7-752707f373ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k" Jul 18 03:15:37.426048 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:37.426006 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c6b1845-d3d3-4fc0-83e7-752707f373ae-maas-api-tls\") pod \"maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k\" (UID: \"1c6b1845-d3d3-4fc0-83e7-752707f373ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k" Jul 18 03:15:37.432538 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:37.432501 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rnmg\" (UniqueName: \"kubernetes.io/projected/1c6b1845-d3d3-4fc0-83e7-752707f373ae-kube-api-access-6rnmg\") pod \"maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k\" (UID: \"1c6b1845-d3d3-4fc0-83e7-752707f373ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k" Jul 18 03:15:37.613553 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:37.613512 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k" Jul 18 03:15:37.776474 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:37.776440 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k"] Jul 18 03:15:37.778907 ip-10-0-140-4 kubenswrapper[2585]: W0718 03:15:37.778865 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c6b1845_d3d3_4fc0_83e7_752707f373ae.slice/crio-a306c57154e1dacf5b4aef033a02c30febc8e2d270d701a4602fcba1d6d08847 WatchSource:0}: Error finding container a306c57154e1dacf5b4aef033a02c30febc8e2d270d701a4602fcba1d6d08847: Status 404 returned error can't find the container with id a306c57154e1dacf5b4aef033a02c30febc8e2d270d701a4602fcba1d6d08847 Jul 18 03:15:38.184402 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:38.184363 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k" event={"ID":"1c6b1845-d3d3-4fc0-83e7-752707f373ae","Type":"ContainerStarted","Data":"a306c57154e1dacf5b4aef033a02c30febc8e2d270d701a4602fcba1d6d08847"} Jul 18 03:15:39.191665 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:39.191619 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k" event={"ID":"1c6b1845-d3d3-4fc0-83e7-752707f373ae","Type":"ContainerStarted","Data":"e270d15954ccc07c6662d06ab36b466ec457e40afdc44b6ffb856067152e864c"} Jul 18 03:15:39.192109 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:39.191692 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k" Jul 18 03:15:39.212145 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:39.212072 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k" podStartSLOduration=1.5492218549999999 podStartE2EDuration="2.212055592s" totalImagesPullingTime="662.833737ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 03:15:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 03:15:39.209543355 +0000 UTC m=+2162.813191054" watchObservedRunningTime="2026-07-18 03:15:39.212055592 +0000 UTC m=+2162.815703278" Jul 18 03:15:45.203711 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:15:45.203676 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k" Jul 18 03:16:05.671105 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:05.671045 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf"] Jul 18 03:16:05.671646 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:05.671341 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf" podUID="7ed6b041-48d6-43e7-82fb-c643cbf25755" containerName="maas-api" containerID="cri-o://92512cff9136863824cb5ce8a1c167992d85e89f3a2cde7d57ab3b1c83995b27" gracePeriod=30 Jul 18 03:16:05.950002 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:05.949966 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf" Jul 18 03:16:06.097723 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:06.097664 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc9n5\" (UniqueName: \"kubernetes.io/projected/7ed6b041-48d6-43e7-82fb-c643cbf25755-kube-api-access-jc9n5\") pod \"7ed6b041-48d6-43e7-82fb-c643cbf25755\" (UID: \"7ed6b041-48d6-43e7-82fb-c643cbf25755\") " Jul 18 03:16:06.097910 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:06.097766 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ed6b041-48d6-43e7-82fb-c643cbf25755-maas-api-tls\") pod \"7ed6b041-48d6-43e7-82fb-c643cbf25755\" (UID: \"7ed6b041-48d6-43e7-82fb-c643cbf25755\") " Jul 18 03:16:06.097910 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:06.097904 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed6b041-48d6-43e7-82fb-c643cbf25755-openshift-service-ca\") pod \"7ed6b041-48d6-43e7-82fb-c643cbf25755\" (UID: \"7ed6b041-48d6-43e7-82fb-c643cbf25755\") " Jul 18 03:16:06.098300 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:06.098267 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ed6b041-48d6-43e7-82fb-c643cbf25755-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7ed6b041-48d6-43e7-82fb-c643cbf25755" (UID: "7ed6b041-48d6-43e7-82fb-c643cbf25755"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 03:16:06.100078 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:06.100046 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ed6b041-48d6-43e7-82fb-c643cbf25755-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7ed6b041-48d6-43e7-82fb-c643cbf25755" (UID: "7ed6b041-48d6-43e7-82fb-c643cbf25755"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 03:16:06.100233 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:06.100088 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed6b041-48d6-43e7-82fb-c643cbf25755-kube-api-access-jc9n5" (OuterVolumeSpecName: "kube-api-access-jc9n5") pod "7ed6b041-48d6-43e7-82fb-c643cbf25755" (UID: "7ed6b041-48d6-43e7-82fb-c643cbf25755"). InnerVolumeSpecName "kube-api-access-jc9n5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 03:16:06.199354 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:06.199253 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed6b041-48d6-43e7-82fb-c643cbf25755-openshift-service-ca\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:16:06.199354 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:06.199290 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc9n5\" (UniqueName: \"kubernetes.io/projected/7ed6b041-48d6-43e7-82fb-c643cbf25755-kube-api-access-jc9n5\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:16:06.199354 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:06.199303 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ed6b041-48d6-43e7-82fb-c643cbf25755-maas-api-tls\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:16:06.339926 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:06.339887 2585 generic.go:356] "Generic (PLEG): container finished" podID="7ed6b041-48d6-43e7-82fb-c643cbf25755" containerID="92512cff9136863824cb5ce8a1c167992d85e89f3a2cde7d57ab3b1c83995b27" exitCode=0 Jul 18 03:16:06.340149 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:06.339935 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf" Jul 18 03:16:06.340149 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:06.339959 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf" event={"ID":"7ed6b041-48d6-43e7-82fb-c643cbf25755","Type":"ContainerDied","Data":"92512cff9136863824cb5ce8a1c167992d85e89f3a2cde7d57ab3b1c83995b27"} Jul 18 03:16:06.340149 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:06.340009 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf" event={"ID":"7ed6b041-48d6-43e7-82fb-c643cbf25755","Type":"ContainerDied","Data":"4f74068dafd1f0f45727a11b994f694147ae99c49ab1fbcf6660379ebe922882"} Jul 18 03:16:06.340149 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:06.340034 2585 scope.go:117] "RemoveContainer" containerID="92512cff9136863824cb5ce8a1c167992d85e89f3a2cde7d57ab3b1c83995b27" Jul 18 03:16:06.350658 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:06.350626 2585 scope.go:117] "RemoveContainer" containerID="92512cff9136863824cb5ce8a1c167992d85e89f3a2cde7d57ab3b1c83995b27" Jul 18 03:16:06.351029 ip-10-0-140-4 kubenswrapper[2585]: E0718 03:16:06.351000 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92512cff9136863824cb5ce8a1c167992d85e89f3a2cde7d57ab3b1c83995b27\": container with ID starting with 92512cff9136863824cb5ce8a1c167992d85e89f3a2cde7d57ab3b1c83995b27 not found: ID does not exist" containerID="92512cff9136863824cb5ce8a1c167992d85e89f3a2cde7d57ab3b1c83995b27" Jul 18 03:16:06.351108 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:06.351039 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92512cff9136863824cb5ce8a1c167992d85e89f3a2cde7d57ab3b1c83995b27"} err="failed to get container status \"92512cff9136863824cb5ce8a1c167992d85e89f3a2cde7d57ab3b1c83995b27\": rpc error: code = NotFound desc = could not find container \"92512cff9136863824cb5ce8a1c167992d85e89f3a2cde7d57ab3b1c83995b27\": container with ID starting with 92512cff9136863824cb5ce8a1c167992d85e89f3a2cde7d57ab3b1c83995b27 not found: ID does not exist" Jul 18 03:16:06.362700 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:06.362654 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf"] Jul 18 03:16:06.368705 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:06.368663 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4e7186-78469c4bb5-bt9lf"] Jul 18 03:16:06.901697 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:06.901653 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ed6b041-48d6-43e7-82fb-c643cbf25755" path="/var/lib/kubelet/pods/7ed6b041-48d6-43e7-82fb-c643cbf25755/volumes" Jul 18 03:16:32.376317 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:32.376278 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k"] Jul 18 03:16:32.376951 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:32.376529 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k" podUID="1c6b1845-d3d3-4fc0-83e7-752707f373ae" containerName="maas-api" containerID="cri-o://e270d15954ccc07c6662d06ab36b466ec457e40afdc44b6ffb856067152e864c" gracePeriod=30 Jul 18 03:16:32.655404 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:32.655367 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k" Jul 18 03:16:32.751971 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:32.751925 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c6b1845-d3d3-4fc0-83e7-752707f373ae-maas-api-tls\") pod \"1c6b1845-d3d3-4fc0-83e7-752707f373ae\" (UID: \"1c6b1845-d3d3-4fc0-83e7-752707f373ae\") " Jul 18 03:16:32.752141 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:32.752032 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rnmg\" (UniqueName: \"kubernetes.io/projected/1c6b1845-d3d3-4fc0-83e7-752707f373ae-kube-api-access-6rnmg\") pod \"1c6b1845-d3d3-4fc0-83e7-752707f373ae\" (UID: \"1c6b1845-d3d3-4fc0-83e7-752707f373ae\") " Jul 18 03:16:32.752141 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:32.752077 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c6b1845-d3d3-4fc0-83e7-752707f373ae-openshift-service-ca\") pod \"1c6b1845-d3d3-4fc0-83e7-752707f373ae\" (UID: \"1c6b1845-d3d3-4fc0-83e7-752707f373ae\") " Jul 18 03:16:32.752550 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:32.752521 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c6b1845-d3d3-4fc0-83e7-752707f373ae-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1c6b1845-d3d3-4fc0-83e7-752707f373ae" (UID: "1c6b1845-d3d3-4fc0-83e7-752707f373ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 03:16:32.754399 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:32.754366 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c6b1845-d3d3-4fc0-83e7-752707f373ae-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1c6b1845-d3d3-4fc0-83e7-752707f373ae" (UID: "1c6b1845-d3d3-4fc0-83e7-752707f373ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 03:16:32.754602 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:32.754408 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c6b1845-d3d3-4fc0-83e7-752707f373ae-kube-api-access-6rnmg" (OuterVolumeSpecName: "kube-api-access-6rnmg") pod "1c6b1845-d3d3-4fc0-83e7-752707f373ae" (UID: "1c6b1845-d3d3-4fc0-83e7-752707f373ae"). InnerVolumeSpecName "kube-api-access-6rnmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 03:16:32.853149 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:32.853089 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c6b1845-d3d3-4fc0-83e7-752707f373ae-maas-api-tls\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:16:32.853149 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:32.853136 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rnmg\" (UniqueName: \"kubernetes.io/projected/1c6b1845-d3d3-4fc0-83e7-752707f373ae-kube-api-access-6rnmg\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:16:32.853149 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:32.853150 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c6b1845-d3d3-4fc0-83e7-752707f373ae-openshift-service-ca\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:16:33.481938 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:33.481902 2585 generic.go:356] "Generic (PLEG): container finished" podID="1c6b1845-d3d3-4fc0-83e7-752707f373ae" containerID="e270d15954ccc07c6662d06ab36b466ec457e40afdc44b6ffb856067152e864c" exitCode=0 Jul 18 03:16:33.482408 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:33.481949 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k" Jul 18 03:16:33.482408 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:33.481976 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k" event={"ID":"1c6b1845-d3d3-4fc0-83e7-752707f373ae","Type":"ContainerDied","Data":"e270d15954ccc07c6662d06ab36b466ec457e40afdc44b6ffb856067152e864c"} Jul 18 03:16:33.482408 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:33.482032 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k" event={"ID":"1c6b1845-d3d3-4fc0-83e7-752707f373ae","Type":"ContainerDied","Data":"a306c57154e1dacf5b4aef033a02c30febc8e2d270d701a4602fcba1d6d08847"} Jul 18 03:16:33.482408 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:33.482045 2585 scope.go:117] "RemoveContainer" containerID="e270d15954ccc07c6662d06ab36b466ec457e40afdc44b6ffb856067152e864c" Jul 18 03:16:33.491550 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:33.491527 2585 scope.go:117] "RemoveContainer" containerID="e270d15954ccc07c6662d06ab36b466ec457e40afdc44b6ffb856067152e864c" Jul 18 03:16:33.491899 ip-10-0-140-4 kubenswrapper[2585]: E0718 03:16:33.491868 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e270d15954ccc07c6662d06ab36b466ec457e40afdc44b6ffb856067152e864c\": container with ID starting with e270d15954ccc07c6662d06ab36b466ec457e40afdc44b6ffb856067152e864c not found: ID does not exist" containerID="e270d15954ccc07c6662d06ab36b466ec457e40afdc44b6ffb856067152e864c" Jul 18 03:16:33.491947 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:33.491911 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e270d15954ccc07c6662d06ab36b466ec457e40afdc44b6ffb856067152e864c"} err="failed to get container status \"e270d15954ccc07c6662d06ab36b466ec457e40afdc44b6ffb856067152e864c\": rpc error: code = NotFound desc = could not find container \"e270d15954ccc07c6662d06ab36b466ec457e40afdc44b6ffb856067152e864c\": container with ID starting with e270d15954ccc07c6662d06ab36b466ec457e40afdc44b6ffb856067152e864c not found: ID does not exist" Jul 18 03:16:33.500962 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:33.500918 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k"] Jul 18 03:16:33.505137 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:33.505092 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bddfdd-84c5b8ddf9-6994k"] Jul 18 03:16:34.902340 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:34.902301 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c6b1845-d3d3-4fc0-83e7-752707f373ae" path="/var/lib/kubelet/pods/1c6b1845-d3d3-4fc0-83e7-752707f373ae/volumes" Jul 18 03:16:47.662355 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:47.662314 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv"] Jul 18 03:16:47.662966 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:47.662943 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ed6b041-48d6-43e7-82fb-c643cbf25755" containerName="maas-api" Jul 18 03:16:47.663061 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:47.662970 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ed6b041-48d6-43e7-82fb-c643cbf25755" containerName="maas-api" Jul 18 03:16:47.663061 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:47.662988 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c6b1845-d3d3-4fc0-83e7-752707f373ae" containerName="maas-api" Jul 18 03:16:47.663061 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:47.662997 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c6b1845-d3d3-4fc0-83e7-752707f373ae" containerName="maas-api" Jul 18 03:16:47.663228 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:47.663095 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ed6b041-48d6-43e7-82fb-c643cbf25755" containerName="maas-api" Jul 18 03:16:47.663228 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:47.663111 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c6b1845-d3d3-4fc0-83e7-752707f373ae" containerName="maas-api" Jul 18 03:16:47.666628 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:47.666593 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv" Jul 18 03:16:47.669161 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:47.669126 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-e88910-serving-cert\"" Jul 18 03:16:47.680336 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:47.680303 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv"] Jul 18 03:16:47.804691 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:47.804640 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f319debc-3ebe-4809-9451-2d7427ec28b0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv\" (UID: \"f319debc-3ebe-4809-9451-2d7427ec28b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv" Jul 18 03:16:47.804940 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:47.804781 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wc8p\" (UniqueName: \"kubernetes.io/projected/f319debc-3ebe-4809-9451-2d7427ec28b0-kube-api-access-6wc8p\") pod \"maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv\" (UID: \"f319debc-3ebe-4809-9451-2d7427ec28b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv" Jul 18 03:16:47.804940 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:47.804861 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f319debc-3ebe-4809-9451-2d7427ec28b0-maas-api-tls\") pod \"maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv\" (UID: \"f319debc-3ebe-4809-9451-2d7427ec28b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv" Jul 18 03:16:47.906494 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:47.906450 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f319debc-3ebe-4809-9451-2d7427ec28b0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv\" (UID: \"f319debc-3ebe-4809-9451-2d7427ec28b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv" Jul 18 03:16:47.906741 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:47.906521 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wc8p\" (UniqueName: \"kubernetes.io/projected/f319debc-3ebe-4809-9451-2d7427ec28b0-kube-api-access-6wc8p\") pod \"maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv\" (UID: \"f319debc-3ebe-4809-9451-2d7427ec28b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv" Jul 18 03:16:47.906741 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:47.906550 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f319debc-3ebe-4809-9451-2d7427ec28b0-maas-api-tls\") pod \"maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv\" (UID: \"f319debc-3ebe-4809-9451-2d7427ec28b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv" Jul 18 03:16:47.906741 ip-10-0-140-4 kubenswrapper[2585]: E0718 03:16:47.906680 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-e88910-serving-cert: secret "maas-api-e2e-shared-a-e88910-serving-cert" not found Jul 18 03:16:47.906929 ip-10-0-140-4 kubenswrapper[2585]: E0718 03:16:47.906763 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f319debc-3ebe-4809-9451-2d7427ec28b0-maas-api-tls podName:f319debc-3ebe-4809-9451-2d7427ec28b0 nodeName:}" failed. No retries permitted until 2026-07-18 03:16:48.406740689 +0000 UTC m=+2232.010388356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f319debc-3ebe-4809-9451-2d7427ec28b0-maas-api-tls") pod "maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv" (UID: "f319debc-3ebe-4809-9451-2d7427ec28b0") : secret "maas-api-e2e-shared-a-e88910-serving-cert" not found Jul 18 03:16:47.907307 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:47.907275 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f319debc-3ebe-4809-9451-2d7427ec28b0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv\" (UID: \"f319debc-3ebe-4809-9451-2d7427ec28b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv" Jul 18 03:16:47.920871 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:47.920783 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wc8p\" (UniqueName: \"kubernetes.io/projected/f319debc-3ebe-4809-9451-2d7427ec28b0-kube-api-access-6wc8p\") pod \"maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv\" (UID: \"f319debc-3ebe-4809-9451-2d7427ec28b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv" Jul 18 03:16:48.412315 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:48.412272 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f319debc-3ebe-4809-9451-2d7427ec28b0-maas-api-tls\") pod \"maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv\" (UID: \"f319debc-3ebe-4809-9451-2d7427ec28b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv" Jul 18 03:16:48.414982 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:48.414948 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f319debc-3ebe-4809-9451-2d7427ec28b0-maas-api-tls\") pod \"maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv\" (UID: \"f319debc-3ebe-4809-9451-2d7427ec28b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv" Jul 18 03:16:48.578412 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:48.578364 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv" Jul 18 03:16:48.734653 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:48.734618 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv"] Jul 18 03:16:48.736886 ip-10-0-140-4 kubenswrapper[2585]: W0718 03:16:48.736842 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf319debc_3ebe_4809_9451_2d7427ec28b0.slice/crio-6cc8eace52bb4c357298a84c52b030c1795aac83fa3a392a5181364bd025a596 WatchSource:0}: Error finding container 6cc8eace52bb4c357298a84c52b030c1795aac83fa3a392a5181364bd025a596: Status 404 returned error can't find the container with id 6cc8eace52bb4c357298a84c52b030c1795aac83fa3a392a5181364bd025a596 Jul 18 03:16:48.738505 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:48.738478 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 03:16:49.569634 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:49.569595 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv" event={"ID":"f319debc-3ebe-4809-9451-2d7427ec28b0","Type":"ContainerStarted","Data":"f4088d19d583c6a3c5f700e3f0284dcb7e6ae8360bc3cb0c2dbe8b94559389ce"} Jul 18 03:16:49.569842 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:49.569661 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv" Jul 18 03:16:49.569842 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:49.569683 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv" event={"ID":"f319debc-3ebe-4809-9451-2d7427ec28b0","Type":"ContainerStarted","Data":"6cc8eace52bb4c357298a84c52b030c1795aac83fa3a392a5181364bd025a596"} Jul 18 03:16:49.589602 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:49.589536 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv" podStartSLOduration=2.158818395 podStartE2EDuration="2.589512901s" totalImagesPullingTime="430.694506ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 03:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 03:16:49.588315693 +0000 UTC m=+2233.191963381" watchObservedRunningTime="2026-07-18 03:16:49.589512901 +0000 UTC m=+2233.193160588" Jul 18 03:16:55.581600 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:55.581566 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv" Jul 18 03:16:55.733718 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:55.733670 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf"] Jul 18 03:16:55.737964 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:55.737931 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf" Jul 18 03:16:55.741178 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:55.741144 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-b50cfd-serving-cert\"" Jul 18 03:16:55.755322 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:55.755286 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf"] Jul 18 03:16:55.775939 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:55.775891 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s894\" (UniqueName: \"kubernetes.io/projected/37b4ed52-b635-4427-8120-2438e8452113-kube-api-access-6s894\") pod \"maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf\" (UID: \"37b4ed52-b635-4427-8120-2438e8452113\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf" Jul 18 03:16:55.776126 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:55.775965 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37b4ed52-b635-4427-8120-2438e8452113-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf\" (UID: \"37b4ed52-b635-4427-8120-2438e8452113\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf" Jul 18 03:16:55.776126 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:55.775990 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37b4ed52-b635-4427-8120-2438e8452113-maas-api-tls\") pod \"maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf\" (UID: \"37b4ed52-b635-4427-8120-2438e8452113\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf" Jul 18 03:16:55.876530 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:55.876424 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37b4ed52-b635-4427-8120-2438e8452113-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf\" (UID: \"37b4ed52-b635-4427-8120-2438e8452113\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf" Jul 18 03:16:55.876530 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:55.876475 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37b4ed52-b635-4427-8120-2438e8452113-maas-api-tls\") pod \"maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf\" (UID: \"37b4ed52-b635-4427-8120-2438e8452113\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf" Jul 18 03:16:55.876530 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:55.876525 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s894\" (UniqueName: \"kubernetes.io/projected/37b4ed52-b635-4427-8120-2438e8452113-kube-api-access-6s894\") pod \"maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf\" (UID: \"37b4ed52-b635-4427-8120-2438e8452113\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf" Jul 18 03:16:55.876789 ip-10-0-140-4 kubenswrapper[2585]: E0718 03:16:55.876645 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-b50cfd-serving-cert: secret "maas-api-e2e-shared-b-b50cfd-serving-cert" not found Jul 18 03:16:55.876949 ip-10-0-140-4 kubenswrapper[2585]: E0718 03:16:55.876929 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37b4ed52-b635-4427-8120-2438e8452113-maas-api-tls podName:37b4ed52-b635-4427-8120-2438e8452113 nodeName:}" failed. No retries permitted until 2026-07-18 03:16:56.376720938 +0000 UTC m=+2239.980368623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/37b4ed52-b635-4427-8120-2438e8452113-maas-api-tls") pod "maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf" (UID: "37b4ed52-b635-4427-8120-2438e8452113") : secret "maas-api-e2e-shared-b-b50cfd-serving-cert" not found Jul 18 03:16:55.877287 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:55.877257 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37b4ed52-b635-4427-8120-2438e8452113-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf\" (UID: \"37b4ed52-b635-4427-8120-2438e8452113\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf" Jul 18 03:16:55.891677 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:55.891640 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s894\" (UniqueName: \"kubernetes.io/projected/37b4ed52-b635-4427-8120-2438e8452113-kube-api-access-6s894\") pod \"maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf\" (UID: \"37b4ed52-b635-4427-8120-2438e8452113\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf" Jul 18 03:16:56.380965 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:56.380911 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37b4ed52-b635-4427-8120-2438e8452113-maas-api-tls\") pod \"maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf\" (UID: \"37b4ed52-b635-4427-8120-2438e8452113\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf" Jul 18 03:16:56.383671 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:56.383640 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37b4ed52-b635-4427-8120-2438e8452113-maas-api-tls\") pod \"maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf\" (UID: \"37b4ed52-b635-4427-8120-2438e8452113\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf" Jul 18 03:16:56.652105 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:56.651973 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf" Jul 18 03:16:56.811799 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:56.811764 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf"] Jul 18 03:16:56.813745 ip-10-0-140-4 kubenswrapper[2585]: W0718 03:16:56.813699 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b4ed52_b635_4427_8120_2438e8452113.slice/crio-903daf64861860c8282e4ec26569870ea29b717fb12a09af0c627ee463bef057 WatchSource:0}: Error finding container 903daf64861860c8282e4ec26569870ea29b717fb12a09af0c627ee463bef057: Status 404 returned error can't find the container with id 903daf64861860c8282e4ec26569870ea29b717fb12a09af0c627ee463bef057 Jul 18 03:16:57.620346 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:57.620299 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf" event={"ID":"37b4ed52-b635-4427-8120-2438e8452113","Type":"ContainerStarted","Data":"632353b23e44ce492f83bd5afbf27832d141c3ea94d9d4125ce0386ffb0a80d4"} Jul 18 03:16:57.620579 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:57.620374 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf" event={"ID":"37b4ed52-b635-4427-8120-2438e8452113","Type":"ContainerStarted","Data":"903daf64861860c8282e4ec26569870ea29b717fb12a09af0c627ee463bef057"} Jul 18 03:16:57.620579 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:57.620520 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf" Jul 18 03:16:57.643939 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:16:57.643870 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf" podStartSLOduration=2.257271844 podStartE2EDuration="2.643852071s" totalImagesPullingTime="386.580227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 03:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 03:16:57.643403954 +0000 UTC m=+2241.247051642" watchObservedRunningTime="2026-07-18 03:16:57.643852071 +0000 UTC m=+2241.247499758" Jul 18 03:17:03.631873 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:17:03.631840 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf" Jul 18 03:19:37.063175 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:19:37.063139 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-krtsg_709575dc-6eae-43e1-9208-cc228795929a/console-operator/1.log" Jul 18 03:19:37.072097 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:19:37.072064 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-krtsg_709575dc-6eae-43e1-9208-cc228795929a/console-operator/1.log" Jul 18 03:23:27.417396 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:27.417344 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv"] Jul 18 03:23:27.418282 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:27.417652 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv" podUID="f319debc-3ebe-4809-9451-2d7427ec28b0" containerName="maas-api" containerID="cri-o://f4088d19d583c6a3c5f700e3f0284dcb7e6ae8360bc3cb0c2dbe8b94559389ce" gracePeriod=30 Jul 18 03:23:27.739764 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:27.739716 2585 generic.go:356] "Generic (PLEG): container finished" podID="f319debc-3ebe-4809-9451-2d7427ec28b0" containerID="f4088d19d583c6a3c5f700e3f0284dcb7e6ae8360bc3cb0c2dbe8b94559389ce" exitCode=0 Jul 18 03:23:27.739915 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:27.739793 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv" event={"ID":"f319debc-3ebe-4809-9451-2d7427ec28b0","Type":"ContainerDied","Data":"f4088d19d583c6a3c5f700e3f0284dcb7e6ae8360bc3cb0c2dbe8b94559389ce"} Jul 18 03:23:27.885030 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:27.884996 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv" Jul 18 03:23:28.066822 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:28.066774 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f319debc-3ebe-4809-9451-2d7427ec28b0-openshift-service-ca\") pod \"f319debc-3ebe-4809-9451-2d7427ec28b0\" (UID: \"f319debc-3ebe-4809-9451-2d7427ec28b0\") " Jul 18 03:23:28.067027 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:28.066848 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f319debc-3ebe-4809-9451-2d7427ec28b0-maas-api-tls\") pod \"f319debc-3ebe-4809-9451-2d7427ec28b0\" (UID: \"f319debc-3ebe-4809-9451-2d7427ec28b0\") " Jul 18 03:23:28.067084 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:28.067023 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wc8p\" (UniqueName: \"kubernetes.io/projected/f319debc-3ebe-4809-9451-2d7427ec28b0-kube-api-access-6wc8p\") pod \"f319debc-3ebe-4809-9451-2d7427ec28b0\" (UID: \"f319debc-3ebe-4809-9451-2d7427ec28b0\") " Jul 18 03:23:28.067280 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:28.067249 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f319debc-3ebe-4809-9451-2d7427ec28b0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f319debc-3ebe-4809-9451-2d7427ec28b0" (UID: "f319debc-3ebe-4809-9451-2d7427ec28b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 03:23:28.067499 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:28.067476 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f319debc-3ebe-4809-9451-2d7427ec28b0-openshift-service-ca\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:23:28.069357 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:28.069323 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f319debc-3ebe-4809-9451-2d7427ec28b0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f319debc-3ebe-4809-9451-2d7427ec28b0" (UID: "f319debc-3ebe-4809-9451-2d7427ec28b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 03:23:28.069357 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:28.069335 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f319debc-3ebe-4809-9451-2d7427ec28b0-kube-api-access-6wc8p" (OuterVolumeSpecName: "kube-api-access-6wc8p") pod "f319debc-3ebe-4809-9451-2d7427ec28b0" (UID: "f319debc-3ebe-4809-9451-2d7427ec28b0"). InnerVolumeSpecName "kube-api-access-6wc8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 03:23:28.168081 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:28.168034 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f319debc-3ebe-4809-9451-2d7427ec28b0-maas-api-tls\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:23:28.168081 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:28.168074 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wc8p\" (UniqueName: \"kubernetes.io/projected/f319debc-3ebe-4809-9451-2d7427ec28b0-kube-api-access-6wc8p\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:23:28.746782 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:28.746750 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv" Jul 18 03:23:28.747285 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:28.746769 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv" event={"ID":"f319debc-3ebe-4809-9451-2d7427ec28b0","Type":"ContainerDied","Data":"6cc8eace52bb4c357298a84c52b030c1795aac83fa3a392a5181364bd025a596"} Jul 18 03:23:28.747285 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:28.746859 2585 scope.go:117] "RemoveContainer" containerID="f4088d19d583c6a3c5f700e3f0284dcb7e6ae8360bc3cb0c2dbe8b94559389ce" Jul 18 03:23:28.771247 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:28.771202 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv"] Jul 18 03:23:28.775681 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:28.775645 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e88910-584c6d69cd-pcwcv"] Jul 18 03:23:28.901958 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:28.901915 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f319debc-3ebe-4809-9451-2d7427ec28b0" path="/var/lib/kubelet/pods/f319debc-3ebe-4809-9451-2d7427ec28b0/volumes" Jul 18 03:23:49.310133 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:49.310067 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf"] Jul 18 03:23:49.311143 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:49.311075 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf" podUID="37b4ed52-b635-4427-8120-2438e8452113" containerName="maas-api" containerID="cri-o://632353b23e44ce492f83bd5afbf27832d141c3ea94d9d4125ce0386ffb0a80d4" gracePeriod=30 Jul 18 03:23:49.577566 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:49.577536 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf" Jul 18 03:23:49.670558 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:49.670520 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37b4ed52-b635-4427-8120-2438e8452113-maas-api-tls\") pod \"37b4ed52-b635-4427-8120-2438e8452113\" (UID: \"37b4ed52-b635-4427-8120-2438e8452113\") " Jul 18 03:23:49.670558 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:49.670573 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37b4ed52-b635-4427-8120-2438e8452113-openshift-service-ca\") pod \"37b4ed52-b635-4427-8120-2438e8452113\" (UID: \"37b4ed52-b635-4427-8120-2438e8452113\") " Jul 18 03:23:49.670839 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:49.670728 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6s894\" (UniqueName: \"kubernetes.io/projected/37b4ed52-b635-4427-8120-2438e8452113-kube-api-access-6s894\") pod \"37b4ed52-b635-4427-8120-2438e8452113\" (UID: \"37b4ed52-b635-4427-8120-2438e8452113\") " Jul 18 03:23:49.671006 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:49.670983 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37b4ed52-b635-4427-8120-2438e8452113-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "37b4ed52-b635-4427-8120-2438e8452113" (UID: "37b4ed52-b635-4427-8120-2438e8452113"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 03:23:49.671097 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:49.671073 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37b4ed52-b635-4427-8120-2438e8452113-openshift-service-ca\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:23:49.672933 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:49.672909 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37b4ed52-b635-4427-8120-2438e8452113-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "37b4ed52-b635-4427-8120-2438e8452113" (UID: "37b4ed52-b635-4427-8120-2438e8452113"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 03:23:49.673152 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:49.673128 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b4ed52-b635-4427-8120-2438e8452113-kube-api-access-6s894" (OuterVolumeSpecName: "kube-api-access-6s894") pod "37b4ed52-b635-4427-8120-2438e8452113" (UID: "37b4ed52-b635-4427-8120-2438e8452113"). InnerVolumeSpecName "kube-api-access-6s894". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 03:23:49.772007 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:49.771951 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s894\" (UniqueName: \"kubernetes.io/projected/37b4ed52-b635-4427-8120-2438e8452113-kube-api-access-6s894\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:23:49.772007 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:49.771997 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37b4ed52-b635-4427-8120-2438e8452113-maas-api-tls\") on node \"ip-10-0-140-4.ec2.internal\" DevicePath \"\"" Jul 18 03:23:49.858484 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:49.858368 2585 generic.go:356] "Generic (PLEG): container finished" podID="37b4ed52-b635-4427-8120-2438e8452113" containerID="632353b23e44ce492f83bd5afbf27832d141c3ea94d9d4125ce0386ffb0a80d4" exitCode=0 Jul 18 03:23:49.858484 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:49.858435 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf" Jul 18 03:23:49.858484 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:49.858458 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf" event={"ID":"37b4ed52-b635-4427-8120-2438e8452113","Type":"ContainerDied","Data":"632353b23e44ce492f83bd5afbf27832d141c3ea94d9d4125ce0386ffb0a80d4"} Jul 18 03:23:49.858742 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:49.858549 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf" event={"ID":"37b4ed52-b635-4427-8120-2438e8452113","Type":"ContainerDied","Data":"903daf64861860c8282e4ec26569870ea29b717fb12a09af0c627ee463bef057"} Jul 18 03:23:49.858742 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:49.858576 2585 scope.go:117] "RemoveContainer" containerID="632353b23e44ce492f83bd5afbf27832d141c3ea94d9d4125ce0386ffb0a80d4" Jul 18 03:23:49.869352 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:49.869329 2585 scope.go:117] "RemoveContainer" containerID="632353b23e44ce492f83bd5afbf27832d141c3ea94d9d4125ce0386ffb0a80d4" Jul 18 03:23:49.869754 ip-10-0-140-4 kubenswrapper[2585]: E0718 03:23:49.869731 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"632353b23e44ce492f83bd5afbf27832d141c3ea94d9d4125ce0386ffb0a80d4\": container with ID starting with 632353b23e44ce492f83bd5afbf27832d141c3ea94d9d4125ce0386ffb0a80d4 not found: ID does not exist" containerID="632353b23e44ce492f83bd5afbf27832d141c3ea94d9d4125ce0386ffb0a80d4" Jul 18 03:23:49.869804 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:49.869764 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"632353b23e44ce492f83bd5afbf27832d141c3ea94d9d4125ce0386ffb0a80d4"} err="failed to get container status \"632353b23e44ce492f83bd5afbf27832d141c3ea94d9d4125ce0386ffb0a80d4\": rpc error: code = NotFound desc = could not find container \"632353b23e44ce492f83bd5afbf27832d141c3ea94d9d4125ce0386ffb0a80d4\": container with ID starting with 632353b23e44ce492f83bd5afbf27832d141c3ea94d9d4125ce0386ffb0a80d4 not found: ID does not exist" Jul 18 03:23:49.880288 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:49.880248 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf"] Jul 18 03:23:49.883798 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:49.883754 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b50cfd-ddd89d488-4qxxf"] Jul 18 03:23:50.901011 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:23:50.900958 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37b4ed52-b635-4427-8120-2438e8452113" path="/var/lib/kubelet/pods/37b4ed52-b635-4427-8120-2438e8452113/volumes" Jul 18 03:24:06.485916 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:06.485876 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-t7wfl_ef50dd81-64dc-4c75-968c-68794de8d2d3/manager/0.log" Jul 18 03:24:06.847596 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:06.847495 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-qw7nf_29bc75bf-af7f-4462-ae2a-df308ef86ec9/manager/0.log" Jul 18 03:24:06.958884 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:06.958854 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f9896c96c-2v8km_cf965887-ba5b-4fef-808f-0006359dd779/copy-manifests/0.log" Jul 18 03:24:06.966303 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:06.966272 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f9896c96c-2v8km_cf965887-ba5b-4fef-808f-0006359dd779/manager/0.log" Jul 18 03:24:07.579311 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:07.579271 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5cdfd7f48c-smpvr_ca4f177c-0e56-47a2-913c-d0668358122c/maas-api/0.log" Jul 18 03:24:07.917451 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:07.917350 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-2txdr_891405f9-75e1-41c6-8874-04c69294207a/postgres/0.log" Jul 18 03:24:09.447832 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:09.447795 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-fhgtx_a3c5af49-8a96-4792-8441-f44c1aa8aca7/manager/0.log" Jul 18 03:24:09.804248 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:09.804208 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-czw74_873dbb75-9c87-4b8f-9b94-c94f274ab3f6/manager/0.log" Jul 18 03:24:10.507619 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:10.507576 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-qjp49_0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98/discovery/0.log" Jul 18 03:24:10.729328 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:10.729295 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58dcbd754f-vt7hf_69eed15a-f1c7-4677-b76a-8aceb91dae02/kube-auth-proxy/0.log" Jul 18 03:24:11.223672 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:11.223625 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-789b768784-4p68t_7ce9775c-ae6f-4891-b9df-58d8a04ec1c8/router/0.log" Jul 18 03:24:11.581305 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:11.581179 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-prvl2_14212281-7994-4954-8913-986f89a0bdaf/main/0.log" Jul 18 03:24:11.694353 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:11.694314 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-szbrt_25ade0b7-80a2-4f51-8c12-2f917c1d164e/main/0.log" Jul 18 03:24:12.035132 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:12.035091 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-zn6pd_e778bb49-179d-4c76-8984-e1bdb8d1cb86/main/0.log" Jul 18 03:24:20.679092 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:20.679046 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mqtcn_d8073815-9d72-4741-8447-ca9fa5dbd049/global-pull-secret-syncer/0.log" Jul 18 03:24:20.850913 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:20.850872 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-h6v9b_76644748-24e4-4b1d-a6b1-68887436724c/konnectivity-agent/0.log" Jul 18 03:24:20.964154 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:20.964027 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-4.ec2.internal_10427b699bfe492315ce6d6f3f8955b6/haproxy/0.log" Jul 18 03:24:26.090582 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:26.090545 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-fhgtx_a3c5af49-8a96-4792-8441-f44c1aa8aca7/manager/0.log" Jul 18 03:24:26.324994 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:26.324947 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-czw74_873dbb75-9c87-4b8f-9b94-c94f274ab3f6/manager/0.log" Jul 18 03:24:28.278415 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:28.278374 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-6l557_2f3a8a13-5e99-4e24-98ee-b112c8b3b784/cluster-monitoring-operator/0.log" Jul 18 03:24:28.565939 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:28.565899 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l279g_e2d82ed0-9996-45da-9757-e1226eb4dff6/node-exporter/0.log" Jul 18 03:24:28.583493 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:28.583449 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l279g_e2d82ed0-9996-45da-9757-e1226eb4dff6/kube-rbac-proxy/0.log" Jul 18 03:24:28.603455 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:28.603419 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l279g_e2d82ed0-9996-45da-9757-e1226eb4dff6/init-textfile/0.log" Jul 18 03:24:30.369630 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:30.369539 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-scxbj_079022d6-f624-49c4-93a6-7bdb4b6a9ffd/networking-console-plugin/0.log" Jul 18 03:24:30.960946 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:30.960914 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-krtsg_709575dc-6eae-43e1-9208-cc228795929a/console-operator/1.log" Jul 18 03:24:30.970995 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:30.970959 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-krtsg_709575dc-6eae-43e1-9208-cc228795929a/console-operator/2.log" Jul 18 03:24:31.182420 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:31.182374 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tn4zf/perf-node-gather-daemonset-77cvq"] Jul 18 03:24:31.182805 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:31.182784 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37b4ed52-b635-4427-8120-2438e8452113" containerName="maas-api" Jul 18 03:24:31.182805 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:31.182805 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="37b4ed52-b635-4427-8120-2438e8452113" containerName="maas-api" Jul 18 03:24:31.182992 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:31.182821 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f319debc-3ebe-4809-9451-2d7427ec28b0" containerName="maas-api" Jul 18 03:24:31.182992 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:31.182830 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f319debc-3ebe-4809-9451-2d7427ec28b0" containerName="maas-api" Jul 18 03:24:31.182992 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:31.182940 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f319debc-3ebe-4809-9451-2d7427ec28b0" containerName="maas-api" Jul 18 03:24:31.182992 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:31.182959 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="37b4ed52-b635-4427-8120-2438e8452113" containerName="maas-api" Jul 18 03:24:31.186356 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:31.186329 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tn4zf/perf-node-gather-daemonset-77cvq" Jul 18 03:24:31.188847 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:31.188821 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tn4zf\"/\"kube-root-ca.crt\"" Jul 18 03:24:31.189007 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:31.188821 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tn4zf\"/\"openshift-service-ca.crt\"" Jul 18 03:24:31.190005 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:31.189984 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tn4zf\"/\"default-dockercfg-glwsz\"" Jul 18 03:24:31.194101 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:31.194062 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tn4zf/perf-node-gather-daemonset-77cvq"] Jul 18 03:24:31.260319 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:31.260273 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/63d9421f-2a1e-49f1-81ef-dd15b4257628-podres\") pod \"perf-node-gather-daemonset-77cvq\" (UID: \"63d9421f-2a1e-49f1-81ef-dd15b4257628\") " pod="openshift-must-gather-tn4zf/perf-node-gather-daemonset-77cvq" Jul 18 03:24:31.260540 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:31.260330 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/63d9421f-2a1e-49f1-81ef-dd15b4257628-proc\") pod \"perf-node-gather-daemonset-77cvq\" (UID: \"63d9421f-2a1e-49f1-81ef-dd15b4257628\") " pod="openshift-must-gather-tn4zf/perf-node-gather-daemonset-77cvq" Jul 18 03:24:31.260540 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:31.260406 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63d9421f-2a1e-49f1-81ef-dd15b4257628-sys\") pod \"perf-node-gather-daemonset-77cvq\" (UID: \"63d9421f-2a1e-49f1-81ef-dd15b4257628\") " pod="openshift-must-gather-tn4zf/perf-node-gather-daemonset-77cvq" Jul 18 03:24:31.260540 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:31.260443 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx8s5\" (UniqueName: \"kubernetes.io/projected/63d9421f-2a1e-49f1-81ef-dd15b4257628-kube-api-access-zx8s5\") pod \"perf-node-gather-daemonset-77cvq\" (UID: \"63d9421f-2a1e-49f1-81ef-dd15b4257628\") " pod="openshift-must-gather-tn4zf/perf-node-gather-daemonset-77cvq" Jul 18 03:24:31.260540 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:31.260475 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63d9421f-2a1e-49f1-81ef-dd15b4257628-lib-modules\") pod \"perf-node-gather-daemonset-77cvq\" (UID: \"63d9421f-2a1e-49f1-81ef-dd15b4257628\") " pod="openshift-must-gather-tn4zf/perf-node-gather-daemonset-77cvq" Jul 18 03:24:31.361434 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:31.361390 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63d9421f-2a1e-49f1-81ef-dd15b4257628-lib-modules\") pod \"perf-node-gather-daemonset-77cvq\" (UID: \"63d9421f-2a1e-49f1-81ef-dd15b4257628\") " pod="openshift-must-gather-tn4zf/perf-node-gather-daemonset-77cvq" Jul 18 03:24:31.361640 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:31.361474 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/63d9421f-2a1e-49f1-81ef-dd15b4257628-podres\") pod \"perf-node-gather-daemonset-77cvq\" (UID: \"63d9421f-2a1e-49f1-81ef-dd15b4257628\") " pod="openshift-must-gather-tn4zf/perf-node-gather-daemonset-77cvq" Jul 18 03:24:31.361640 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:31.361506 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/63d9421f-2a1e-49f1-81ef-dd15b4257628-proc\") pod \"perf-node-gather-daemonset-77cvq\" (UID: \"63d9421f-2a1e-49f1-81ef-dd15b4257628\") " pod="openshift-must-gather-tn4zf/perf-node-gather-daemonset-77cvq" Jul 18 03:24:31.361640 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:31.361530 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63d9421f-2a1e-49f1-81ef-dd15b4257628-sys\") pod \"perf-node-gather-daemonset-77cvq\" (UID: \"63d9421f-2a1e-49f1-81ef-dd15b4257628\") " pod="openshift-must-gather-tn4zf/perf-node-gather-daemonset-77cvq" Jul 18 03:24:31.361640 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:31.361551 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx8s5\" (UniqueName: \"kubernetes.io/projected/63d9421f-2a1e-49f1-81ef-dd15b4257628-kube-api-access-zx8s5\") pod \"perf-node-gather-daemonset-77cvq\" (UID: \"63d9421f-2a1e-49f1-81ef-dd15b4257628\") " pod="openshift-must-gather-tn4zf/perf-node-gather-daemonset-77cvq" Jul 18 03:24:31.361640 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:31.361615 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63d9421f-2a1e-49f1-81ef-dd15b4257628-lib-modules\") pod \"perf-node-gather-daemonset-77cvq\" (UID: \"63d9421f-2a1e-49f1-81ef-dd15b4257628\") " pod="openshift-must-gather-tn4zf/perf-node-gather-daemonset-77cvq" Jul 18 03:24:31.361809 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:31.361635 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63d9421f-2a1e-49f1-81ef-dd15b4257628-sys\") pod \"perf-node-gather-daemonset-77cvq\" (UID: \"63d9421f-2a1e-49f1-81ef-dd15b4257628\") " pod="openshift-must-gather-tn4zf/perf-node-gather-daemonset-77cvq" Jul 18 03:24:31.361809 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:31.361635 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/63d9421f-2a1e-49f1-81ef-dd15b4257628-podres\") pod \"perf-node-gather-daemonset-77cvq\" (UID: \"63d9421f-2a1e-49f1-81ef-dd15b4257628\") " pod="openshift-must-gather-tn4zf/perf-node-gather-daemonset-77cvq" Jul 18 03:24:31.361809 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:31.361615 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/63d9421f-2a1e-49f1-81ef-dd15b4257628-proc\") pod \"perf-node-gather-daemonset-77cvq\" (UID: \"63d9421f-2a1e-49f1-81ef-dd15b4257628\") " pod="openshift-must-gather-tn4zf/perf-node-gather-daemonset-77cvq" Jul 18 03:24:31.369941 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:31.369887 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx8s5\" (UniqueName: \"kubernetes.io/projected/63d9421f-2a1e-49f1-81ef-dd15b4257628-kube-api-access-zx8s5\") pod \"perf-node-gather-daemonset-77cvq\" (UID: \"63d9421f-2a1e-49f1-81ef-dd15b4257628\") " pod="openshift-must-gather-tn4zf/perf-node-gather-daemonset-77cvq" Jul 18 03:24:31.499407 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:31.499357 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tn4zf/perf-node-gather-daemonset-77cvq" Jul 18 03:24:31.653685 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:31.653649 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tn4zf/perf-node-gather-daemonset-77cvq"] Jul 18 03:24:31.656613 ip-10-0-140-4 kubenswrapper[2585]: W0718 03:24:31.656578 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod63d9421f_2a1e_49f1_81ef_dd15b4257628.slice/crio-982e442f54315503625c260210b12d49917da6261a65ac1a458533bfb7de9ad4 WatchSource:0}: Error finding container 982e442f54315503625c260210b12d49917da6261a65ac1a458533bfb7de9ad4: Status 404 returned error can't find the container with id 982e442f54315503625c260210b12d49917da6261a65ac1a458533bfb7de9ad4 Jul 18 03:24:32.078269 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:32.078227 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tn4zf/perf-node-gather-daemonset-77cvq" event={"ID":"63d9421f-2a1e-49f1-81ef-dd15b4257628","Type":"ContainerStarted","Data":"a57331c5cfe9917f4249b8d25ae4b1e95b2b7f7718edd09a2ee064d5793665c1"} Jul 18 03:24:32.078527 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:32.078292 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tn4zf/perf-node-gather-daemonset-77cvq" event={"ID":"63d9421f-2a1e-49f1-81ef-dd15b4257628","Type":"ContainerStarted","Data":"982e442f54315503625c260210b12d49917da6261a65ac1a458533bfb7de9ad4"} Jul 18 03:24:32.078527 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:32.078316 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tn4zf/perf-node-gather-daemonset-77cvq" Jul 18 03:24:32.095161 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:32.095098 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tn4zf/perf-node-gather-daemonset-77cvq" podStartSLOduration=1.095077629 podStartE2EDuration="1.095077629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 03:24:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 03:24:32.093376667 +0000 UTC m=+2695.697024354" watchObservedRunningTime="2026-07-18 03:24:32.095077629 +0000 UTC m=+2695.698725316" Jul 18 03:24:32.197987 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:32.197935 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-44wcx_9d34d35f-234b-4902-8a99-bc58ca2185de/volume-data-source-validator/0.log" Jul 18 03:24:33.167573 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:33.167541 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w4wvr_7e95a777-c662-4a6a-bac1-9fa079e5914c/dns/0.log" Jul 18 03:24:33.184738 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:33.184704 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w4wvr_7e95a777-c662-4a6a-bac1-9fa079e5914c/kube-rbac-proxy/0.log" Jul 18 03:24:33.285910 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:33.285881 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tlhsl_7321f9a8-9e03-4dff-a574-0175b0c89260/dns-node-resolver/0.log" Jul 18 03:24:33.821909 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:33.821868 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-625jl_1b59997b-ff58-43bc-b393-a6c42f6f1ed4/node-ca/0.log" Jul 18 03:24:34.988768 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:34.988725 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-qjp49_0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98/discovery/0.log" Jul 18 03:24:35.030334 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:35.030293 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58dcbd754f-vt7hf_69eed15a-f1c7-4677-b76a-8aceb91dae02/kube-auth-proxy/0.log" Jul 18 03:24:35.214649 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:35.214613 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-789b768784-4p68t_7ce9775c-ae6f-4891-b9df-58d8a04ec1c8/router/0.log" Jul 18 03:24:35.923432 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:35.923396 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vbsgm_2aceb1e5-bf1c-4344-aaa0-0e3f9fd11ab5/serve-healthcheck-canary/0.log" Jul 18 03:24:36.359481 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:36.359439 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-d5t6p_52dc3a30-b4eb-40ce-8380-9fc3a18db5df/insights-operator/0.log" Jul 18 03:24:36.361292 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:36.361256 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-d5t6p_52dc3a30-b4eb-40ce-8380-9fc3a18db5df/insights-operator/1.log" Jul 18 03:24:36.379707 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:36.379673 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fvt9x_f7b220b6-d648-4f31-b11b-2d3d7be9a13a/kube-rbac-proxy/0.log" Jul 18 03:24:36.397682 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:36.397649 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fvt9x_f7b220b6-d648-4f31-b11b-2d3d7be9a13a/exporter/0.log" Jul 18 03:24:36.416429 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:36.416394 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fvt9x_f7b220b6-d648-4f31-b11b-2d3d7be9a13a/extractor/0.log" Jul 18 03:24:37.104440 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:37.104405 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-krtsg_709575dc-6eae-43e1-9208-cc228795929a/console-operator/1.log" Jul 18 03:24:37.113016 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:37.112981 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-krtsg_709575dc-6eae-43e1-9208-cc228795929a/console-operator/1.log" Jul 18 03:24:38.094768 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:38.094737 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tn4zf/perf-node-gather-daemonset-77cvq" Jul 18 03:24:38.933977 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:38.933931 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-t7wfl_ef50dd81-64dc-4c75-968c-68794de8d2d3/manager/0.log" Jul 18 03:24:39.253821 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:39.253311 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-qw7nf_29bc75bf-af7f-4462-ae2a-df308ef86ec9/manager/0.log" Jul 18 03:24:39.284717 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:39.284675 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f9896c96c-2v8km_cf965887-ba5b-4fef-808f-0006359dd779/manager/0.log" Jul 18 03:24:39.304355 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:39.304324 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f9896c96c-2v8km_cf965887-ba5b-4fef-808f-0006359dd779/copy-manifests/0.log" Jul 18 03:24:41.810784 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:41.810747 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-qjp49_0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98/discovery/0.log" Jul 18 03:24:41.835368 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:41.835324 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58dcbd754f-vt7hf_69eed15a-f1c7-4677-b76a-8aceb91dae02/kube-auth-proxy/0.log" Jul 18 03:24:41.998989 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:41.998953 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-789b768784-4p68t_7ce9775c-ae6f-4891-b9df-58d8a04ec1c8/router/0.log" Jul 18 03:24:43.482145 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:43.482101 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-qjp49_0452e2c7-0a7a-4f35-aacc-c0a49ca8fe98/discovery/0.log" Jul 18 03:24:43.515403 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:43.515365 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58dcbd754f-vt7hf_69eed15a-f1c7-4677-b76a-8aceb91dae02/kube-auth-proxy/0.log" Jul 18 03:24:43.679162 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:43.679124 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-789b768784-4p68t_7ce9775c-ae6f-4891-b9df-58d8a04ec1c8/router/0.log" Jul 18 03:24:46.333313 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:46.333263 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-c4l7k_e05161ba-27ca-4679-9439-bad0efa4500f/kube-storage-version-migrator-operator/1.log" Jul 18 03:24:46.334980 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:46.334945 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-c4l7k_e05161ba-27ca-4679-9439-bad0efa4500f/kube-storage-version-migrator-operator/0.log" Jul 18 03:24:47.457315 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:47.457281 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8b2g5_d56b8798-4bc3-4101-b769-c129c5097167/kube-multus/0.log" Jul 18 03:24:47.750768 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:47.750739 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xvpps_35cb46c9-0c8a-4562-a059-521ad01f2552/kube-multus-additional-cni-plugins/0.log" Jul 18 03:24:47.769290 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:47.769252 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xvpps_35cb46c9-0c8a-4562-a059-521ad01f2552/egress-router-binary-copy/0.log" Jul 18 03:24:47.787507 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:47.787475 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xvpps_35cb46c9-0c8a-4562-a059-521ad01f2552/cni-plugins/0.log" Jul 18 03:24:47.805838 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:47.805791 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xvpps_35cb46c9-0c8a-4562-a059-521ad01f2552/bond-cni-plugin/0.log" Jul 18 03:24:47.823721 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:47.823690 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xvpps_35cb46c9-0c8a-4562-a059-521ad01f2552/routeoverride-cni/0.log" Jul 18 03:24:47.846838 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:47.846811 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xvpps_35cb46c9-0c8a-4562-a059-521ad01f2552/whereabouts-cni-bincopy/0.log" Jul 18 03:24:47.868423 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:47.868339 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xvpps_35cb46c9-0c8a-4562-a059-521ad01f2552/whereabouts-cni/0.log" Jul 18 03:24:48.006139 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:48.006042 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nht5x_870f3a38-67e6-42ea-9398-f12eeb350f5d/network-metrics-daemon/0.log" Jul 18 03:24:48.021422 ip-10-0-140-4 kubenswrapper[2585]: I0718 03:24:48.021385 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nht5x_870f3a38-67e6-42ea-9398-f12eeb350f5d/kube-rbac-proxy/0.log"