Jul 23 17:42:36.877907 ip-10-0-137-82 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 23 17:42:36.877921 ip-10-0-137-82 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 23 17:42:36.877932 ip-10-0-137-82 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 23 17:42:36.878336 ip-10-0-137-82 systemd[1]: Failed to start Kubernetes Kubelet. Jul 23 17:42:45.605685 ip-10-0-137-82 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Jul 23 17:42:45.605706 ip-10-0-137-82 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 312157c94a6f46988b674e8b3bca25ba -- Jul 23 17:44:15.791886 ip-10-0-137-82 systemd[1]: Starting Kubernetes Kubelet... Jul 23 17:44:16.206781 ip-10-0-137-82 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 17:44:16.206781 ip-10-0-137-82 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 23 17:44:16.206781 ip-10-0-137-82 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 17:44:16.206781 ip-10-0-137-82 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 23 17:44:16.206781 ip-10-0-137-82 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 17:44:16.208766 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.208679 2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 23 17:44:16.212175 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212160 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 17:44:16.212175 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212176 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 17:44:16.212242 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212180 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 17:44:16.212242 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212184 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 17:44:16.212242 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212187 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 23 17:44:16.212242 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212190 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 17:44:16.212242 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212193 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 17:44:16.212242 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212196 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 17:44:16.212242 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212199 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 17:44:16.212242 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212203 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 17:44:16.212242 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212205 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 17:44:16.212242 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212208 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 17:44:16.212242 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212210 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 17:44:16.212242 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212213 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 17:44:16.212242 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212216 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 17:44:16.212242 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212218 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 17:44:16.212242 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212221 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 17:44:16.212242 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212224 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 17:44:16.212242 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212226 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 17:44:16.212242 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212229 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 17:44:16.212242 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212232 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 17:44:16.212242 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212235 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 17:44:16.212722 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212239 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 17:44:16.212722 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212243 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 17:44:16.212722 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212248 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 17:44:16.212722 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212252 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 17:44:16.212722 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212255 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 17:44:16.212722 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212258 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 17:44:16.212722 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212261 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 17:44:16.212722 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212264 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 17:44:16.212722 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212266 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 17:44:16.212722 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212269 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 17:44:16.212722 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212272 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 17:44:16.212722 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212274 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 17:44:16.212722 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212276 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 17:44:16.212722 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212279 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 17:44:16.212722 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212282 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 17:44:16.212722 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212284 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 17:44:16.212722 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212287 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 17:44:16.212722 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212289 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 17:44:16.212722 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212292 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 17:44:16.213164 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212294 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 17:44:16.213164 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212297 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 17:44:16.213164 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212300 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 17:44:16.213164 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212302 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 17:44:16.213164 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212305 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 17:44:16.213164 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212307 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 17:44:16.213164 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212310 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 17:44:16.213164 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212312 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 17:44:16.213164 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212315 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 17:44:16.213164 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212317 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 17:44:16.213164 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212320 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 17:44:16.213164 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212323 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 17:44:16.213164 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212325 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 17:44:16.213164 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212328 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 17:44:16.213164 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212331 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 17:44:16.213164 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212334 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 17:44:16.213164 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212336 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 17:44:16.213164 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212339 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 17:44:16.213164 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212341 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 17:44:16.213164 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212344 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 17:44:16.213659 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212346 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 17:44:16.213659 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212349 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 17:44:16.213659 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212351 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 17:44:16.213659 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212354 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 17:44:16.213659 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212356 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 17:44:16.213659 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212359 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 17:44:16.213659 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212361 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 17:44:16.213659 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212364 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 17:44:16.213659 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212366 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 17:44:16.213659 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212369 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 17:44:16.213659 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212372 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 17:44:16.213659 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212374 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 17:44:16.213659 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212376 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 17:44:16.213659 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212379 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 17:44:16.213659 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212382 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 17:44:16.213659 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212384 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 17:44:16.213659 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212387 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 17:44:16.213659 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212389 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 17:44:16.213659 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212392 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 17:44:16.214211 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212394 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 17:44:16.214211 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212397 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 17:44:16.214211 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212399 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 17:44:16.214211 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212403 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 17:44:16.214211 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212406 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 17:44:16.214211 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212409 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 17:44:16.214211 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212411 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 17:44:16.214211 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212418 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 17:44:16.214211 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212421 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 17:44:16.214211 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212425 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 17:44:16.214211 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212429 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 17:44:16.214211 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212431 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 17:44:16.214211 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212434 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 17:44:16.214211 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212436 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 17:44:16.214211 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212439 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 17:44:16.214211 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212442 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 17:44:16.214211 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212444 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 17:44:16.214211 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212447 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 17:44:16.214211 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.212450 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 17:44:16.214940 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213146 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 17:44:16.214940 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213153 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 17:44:16.214940 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213155 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 17:44:16.214940 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213159 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 17:44:16.214940 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213161 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 17:44:16.214940 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213164 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 17:44:16.214940 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213166 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 17:44:16.214940 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213169 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 17:44:16.214940 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213172 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 17:44:16.214940 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213175 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 17:44:16.214940 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213178 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 17:44:16.214940 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213181 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 17:44:16.214940 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213184 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 17:44:16.214940 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213186 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 17:44:16.214940 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213189 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 17:44:16.214940 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213192 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 17:44:16.214940 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213195 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 17:44:16.214940 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213197 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 17:44:16.214940 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213200 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 17:44:16.214940 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213203 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 17:44:16.215442 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213205 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 17:44:16.215442 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213208 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 17:44:16.215442 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213211 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 17:44:16.215442 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213214 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 17:44:16.215442 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213216 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 17:44:16.215442 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213219 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 17:44:16.215442 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213222 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 17:44:16.215442 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213224 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 17:44:16.215442 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213226 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 17:44:16.215442 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213229 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 17:44:16.215442 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213232 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 17:44:16.215442 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213234 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 17:44:16.215442 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213237 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 17:44:16.215442 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213241 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 17:44:16.215442 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213244 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 17:44:16.215442 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213247 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 17:44:16.215442 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213250 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 17:44:16.215442 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213253 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 23 17:44:16.215442 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213255 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 17:44:16.215442 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213259 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 17:44:16.215977 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213261 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 17:44:16.215977 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213264 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 17:44:16.215977 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213268 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 17:44:16.215977 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213271 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 17:44:16.215977 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213274 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 17:44:16.215977 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213277 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 17:44:16.215977 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213279 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 17:44:16.215977 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213282 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 17:44:16.215977 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213284 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 17:44:16.215977 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213286 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 17:44:16.215977 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213289 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 17:44:16.215977 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213292 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 17:44:16.215977 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213295 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 17:44:16.215977 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213299 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 17:44:16.215977 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213303 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 17:44:16.215977 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213305 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 17:44:16.215977 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213308 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 17:44:16.215977 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213311 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 17:44:16.215977 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213313 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 17:44:16.215977 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213315 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 17:44:16.216458 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213318 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 17:44:16.216458 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213320 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 17:44:16.216458 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213323 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 17:44:16.216458 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213325 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 17:44:16.216458 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213328 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 17:44:16.216458 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213330 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 17:44:16.216458 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213333 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 17:44:16.216458 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213336 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 17:44:16.216458 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213338 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 17:44:16.216458 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213340 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 17:44:16.216458 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213343 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 17:44:16.216458 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213346 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 17:44:16.216458 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213349 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 17:44:16.216458 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213351 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 17:44:16.216458 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213354 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 17:44:16.216458 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213357 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 17:44:16.216458 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213359 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 17:44:16.216458 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213362 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 17:44:16.216458 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213365 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 17:44:16.216931 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213368 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 17:44:16.216931 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213372 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 17:44:16.216931 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213376 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 17:44:16.216931 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213379 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 17:44:16.216931 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213382 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 17:44:16.216931 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213384 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 17:44:16.216931 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213387 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 17:44:16.216931 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213389 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 17:44:16.216931 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213392 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 17:44:16.216931 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213395 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 17:44:16.216931 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213397 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 17:44:16.216931 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213400 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 17:44:16.216931 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213402 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 17:44:16.216931 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213405 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 17:44:16.216931 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213407 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 17:44:16.216931 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213410 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 17:44:16.216931 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213413 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 17:44:16.216931 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213415 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 17:44:16.216931 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213418 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 17:44:16.217395 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213421 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 17:44:16.217395 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213798 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 17:44:16.217395 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213803 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 17:44:16.217395 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213806 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 17:44:16.217395 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213809 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 17:44:16.217395 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213811 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 17:44:16.217395 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213814 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 17:44:16.217395 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213817 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 17:44:16.217395 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213820 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 17:44:16.217395 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213822 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 17:44:16.217395 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213825 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 17:44:16.217395 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213827 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 17:44:16.217395 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213830 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 17:44:16.217395 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213832 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 17:44:16.217395 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213835 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 23 17:44:16.217395 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213837 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 17:44:16.217395 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213840 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 17:44:16.217395 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213842 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 17:44:16.217395 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213845 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 17:44:16.217857 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213848 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 17:44:16.217857 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213850 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 17:44:16.217857 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213853 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 17:44:16.217857 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213856 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 17:44:16.217857 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213858 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 17:44:16.217857 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213861 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 17:44:16.217857 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213864 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 17:44:16.217857 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213866 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 17:44:16.217857 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213868 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 17:44:16.217857 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213871 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 17:44:16.217857 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213873 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 17:44:16.217857 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213876 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 17:44:16.217857 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213878 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 17:44:16.217857 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213881 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 17:44:16.217857 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213884 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 17:44:16.217857 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213886 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 17:44:16.217857 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213889 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 17:44:16.217857 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213893 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 17:44:16.217857 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213895 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 17:44:16.217857 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213898 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 17:44:16.218358 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213900 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 17:44:16.218358 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.213903 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 17:44:16.218358 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215464 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 17:44:16.218358 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215477 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 17:44:16.218358 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215480 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 17:44:16.218358 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215484 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 17:44:16.218358 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215487 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 17:44:16.218358 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215490 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 17:44:16.218358 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215493 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 17:44:16.218358 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215495 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 17:44:16.218358 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215501 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 17:44:16.218358 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215505 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 17:44:16.218358 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215510 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 17:44:16.218358 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215514 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 17:44:16.218358 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215545 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 17:44:16.218358 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215560 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 17:44:16.218358 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215565 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 17:44:16.218358 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215569 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 17:44:16.218358 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215573 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 17:44:16.218862 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215576 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 17:44:16.218862 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215580 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 17:44:16.218862 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215583 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 17:44:16.218862 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215587 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 17:44:16.218862 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215590 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 17:44:16.218862 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215593 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 17:44:16.218862 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215627 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 17:44:16.218862 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215667 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 17:44:16.218862 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215672 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 17:44:16.218862 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215781 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 17:44:16.218862 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215789 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 17:44:16.218862 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215792 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 17:44:16.218862 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215795 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 17:44:16.218862 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215798 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 17:44:16.218862 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215800 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 17:44:16.218862 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215803 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 17:44:16.218862 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215806 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 17:44:16.218862 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215809 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 17:44:16.218862 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215811 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 17:44:16.218862 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215814 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 17:44:16.218862 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215817 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 17:44:16.219354 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215819 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 17:44:16.219354 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215822 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 17:44:16.219354 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215825 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 17:44:16.219354 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215827 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 17:44:16.219354 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215832 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 17:44:16.219354 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215835 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 17:44:16.219354 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215838 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 17:44:16.219354 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215840 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 17:44:16.219354 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215845 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 17:44:16.219354 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215851 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 17:44:16.219354 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215855 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 17:44:16.219354 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215858 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 17:44:16.219354 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215861 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 17:44:16.219354 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215864 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 17:44:16.219354 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215867 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 17:44:16.219354 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215870 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 17:44:16.219354 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215873 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 17:44:16.219354 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215875 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 17:44:16.219354 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215878 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 17:44:16.219822 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215881 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 17:44:16.219822 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.215883 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 17:44:16.219822 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.215964 2565 flags.go:64] FLAG: --address="0.0.0.0" Jul 23 17:44:16.219822 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.215971 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 23 17:44:16.219822 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.215978 2565 flags.go:64] FLAG: --anonymous-auth="true" Jul 23 17:44:16.219822 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.215983 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 23 17:44:16.219822 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.215987 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 23 17:44:16.219822 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.215990 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 23 17:44:16.219822 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.215994 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 23 17:44:16.219822 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.215999 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 23 17:44:16.219822 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216002 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 23 17:44:16.219822 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216005 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 23 17:44:16.219822 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216008 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 23 17:44:16.219822 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216011 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 23 17:44:16.219822 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216015 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 23 17:44:16.219822 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216018 2565 flags.go:64] FLAG: --cgroup-root="" Jul 23 17:44:16.219822 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216021 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 23 17:44:16.219822 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216023 2565 flags.go:64] FLAG: --client-ca-file="" Jul 23 17:44:16.219822 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216026 2565 flags.go:64] FLAG: --cloud-provider="external" Jul 23 17:44:16.219822 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216029 2565 flags.go:64] FLAG: --cluster-dns="[]" Jul 23 17:44:16.219822 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216035 2565 flags.go:64] FLAG: --cluster-domain="" Jul 23 17:44:16.219822 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216038 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 23 17:44:16.219822 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216041 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 23 17:44:16.220394 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216044 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 23 17:44:16.220394 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216048 2565 flags.go:64] FLAG: --container-log-max-files="5" Jul 23 17:44:16.220394 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216052 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 23 17:44:16.220394 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216055 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 23 17:44:16.220394 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216058 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 23 17:44:16.220394 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216061 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 23 17:44:16.220394 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216064 2565 flags.go:64] FLAG: --contention-profiling="false" Jul 23 17:44:16.220394 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216067 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 23 17:44:16.220394 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216071 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 23 17:44:16.220394 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216074 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 23 17:44:16.220394 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216077 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 23 17:44:16.220394 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216081 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 23 17:44:16.220394 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216085 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 23 17:44:16.220394 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216087 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 23 17:44:16.220394 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216090 2565 flags.go:64] FLAG: --enable-load-reader="false" Jul 23 17:44:16.220394 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216093 2565 flags.go:64] FLAG: --enable-server="true" Jul 23 17:44:16.220394 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216096 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 23 17:44:16.220394 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216100 2565 flags.go:64] FLAG: --event-burst="100" Jul 23 17:44:16.220394 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216103 2565 flags.go:64] FLAG: --event-qps="50" Jul 23 17:44:16.220394 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216106 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 23 17:44:16.220394 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216109 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 23 17:44:16.220394 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216112 2565 flags.go:64] FLAG: --eviction-hard="" Jul 23 17:44:16.220394 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216116 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 23 17:44:16.220394 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216119 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 23 17:44:16.220960 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216122 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 23 17:44:16.220960 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216125 2565 flags.go:64] FLAG: --eviction-soft="" Jul 23 17:44:16.220960 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216128 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 23 17:44:16.220960 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216131 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 23 17:44:16.220960 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216133 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 23 17:44:16.220960 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216137 2565 flags.go:64] FLAG: --experimental-mounter-path="" Jul 23 17:44:16.220960 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216139 2565 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 23 17:44:16.220960 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216143 2565 flags.go:64] FLAG: --fail-swap-on="true" Jul 23 17:44:16.220960 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216146 2565 flags.go:64] FLAG: --feature-gates="" Jul 23 17:44:16.220960 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216150 2565 flags.go:64] FLAG: --file-check-frequency="20s" Jul 23 17:44:16.220960 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216153 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 23 17:44:16.220960 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216156 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 23 17:44:16.220960 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216160 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 23 17:44:16.220960 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216163 2565 flags.go:64] FLAG: --healthz-port="10248" Jul 23 17:44:16.220960 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216166 2565 flags.go:64] FLAG: --help="false" Jul 23 17:44:16.220960 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216169 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-137-82.ec2.internal" Jul 23 17:44:16.220960 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216172 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 23 17:44:16.220960 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216175 2565 flags.go:64] FLAG: --http-check-frequency="20s" Jul 23 17:44:16.220960 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216177 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 23 17:44:16.220960 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216181 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 23 17:44:16.220960 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216184 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 23 17:44:16.220960 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216187 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 23 17:44:16.220960 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216190 2565 flags.go:64] FLAG: --image-service-endpoint="" Jul 23 17:44:16.221527 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216192 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 23 17:44:16.221527 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216195 2565 flags.go:64] FLAG: --kube-api-burst="100" Jul 23 17:44:16.221527 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216198 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 23 17:44:16.221527 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216201 2565 flags.go:64] FLAG: --kube-api-qps="50" Jul 23 17:44:16.221527 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216204 2565 flags.go:64] FLAG: --kube-reserved="" Jul 23 17:44:16.221527 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216207 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 23 17:44:16.221527 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216209 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 23 17:44:16.221527 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216212 2565 flags.go:64] FLAG: --kubelet-cgroups="" Jul 23 17:44:16.221527 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216215 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 23 17:44:16.221527 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216218 2565 flags.go:64] FLAG: --lock-file="" Jul 23 17:44:16.221527 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216220 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 23 17:44:16.221527 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216223 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 23 17:44:16.221527 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216226 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 23 17:44:16.221527 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216231 2565 flags.go:64] FLAG: --log-json-split-stream="false" Jul 23 17:44:16.221527 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216234 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 23 17:44:16.221527 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216237 2565 flags.go:64] FLAG: --log-text-split-stream="false" Jul 23 17:44:16.221527 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216242 2565 flags.go:64] FLAG: --logging-format="text" Jul 23 17:44:16.221527 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216245 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 23 17:44:16.221527 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216248 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 23 17:44:16.221527 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216251 2565 flags.go:64] FLAG: --manifest-url="" Jul 23 17:44:16.221527 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216255 2565 flags.go:64] FLAG: --manifest-url-header="" Jul 23 17:44:16.221527 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216259 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 23 17:44:16.221527 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216263 2565 flags.go:64] FLAG: --max-open-files="1000000" Jul 23 17:44:16.221527 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216267 2565 flags.go:64] FLAG: --max-pods="110" Jul 23 17:44:16.221527 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216270 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 23 17:44:16.222142 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216273 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 23 17:44:16.222142 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216276 2565 flags.go:64] FLAG: --memory-manager-policy="None" Jul 23 17:44:16.222142 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216279 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 23 17:44:16.222142 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216282 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 23 17:44:16.222142 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216285 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 23 17:44:16.222142 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216288 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 23 17:44:16.222142 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216295 2565 flags.go:64] FLAG: --node-status-max-images="50" Jul 23 17:44:16.222142 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216298 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 23 17:44:16.222142 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216301 2565 flags.go:64] FLAG: --oom-score-adj="-999" Jul 23 17:44:16.222142 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216304 2565 flags.go:64] FLAG: --pod-cidr="" Jul 23 17:44:16.222142 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216307 2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 23 17:44:16.222142 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216312 2565 flags.go:64] FLAG: --pod-manifest-path="" Jul 23 17:44:16.222142 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216315 2565 flags.go:64] FLAG: --pod-max-pids="-1" Jul 23 17:44:16.222142 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216318 2565 flags.go:64] FLAG: --pods-per-core="0" Jul 23 17:44:16.222142 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216321 2565 flags.go:64] FLAG: --port="10250" Jul 23 17:44:16.222142 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216324 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 23 17:44:16.222142 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216327 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08fc1f640ca27079e" Jul 23 17:44:16.222142 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216330 2565 flags.go:64] FLAG: --qos-reserved="" Jul 23 17:44:16.222142 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216333 2565 flags.go:64] FLAG: --read-only-port="10255" Jul 23 17:44:16.222142 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216336 2565 flags.go:64] FLAG: --register-node="true" Jul 23 17:44:16.222142 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216339 2565 flags.go:64] FLAG: --register-with-taints="" Jul 23 17:44:16.222142 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216342 2565 flags.go:64] FLAG: --registry-burst="10" Jul 23 17:44:16.222142 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216345 2565 flags.go:64] FLAG: --registry-qps="5" Jul 23 17:44:16.222142 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216348 2565 flags.go:64] FLAG: --reserved-cpus="" Jul 23 17:44:16.222780 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216353 2565 flags.go:64] FLAG: --reserved-memory="" Jul 23 17:44:16.222780 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216356 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 23 17:44:16.222780 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216360 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 23 17:44:16.222780 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216363 2565 flags.go:64] FLAG: --rotate-certificates="false" Jul 23 17:44:16.222780 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216366 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 23 17:44:16.222780 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216368 2565 flags.go:64] FLAG: --runonce="false" Jul 23 17:44:16.222780 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216372 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 23 17:44:16.222780 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216376 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 23 17:44:16.222780 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216379 2565 flags.go:64] FLAG: --seccomp-default="false" Jul 23 17:44:16.222780 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216381 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 23 17:44:16.222780 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216385 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 23 17:44:16.222780 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216387 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 23 17:44:16.222780 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216390 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 23 17:44:16.222780 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216394 2565 flags.go:64] FLAG: --storage-driver-password="root" Jul 23 17:44:16.222780 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216396 2565 flags.go:64] FLAG: --storage-driver-secure="false" Jul 23 17:44:16.222780 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216399 2565 flags.go:64] FLAG: --storage-driver-table="stats" Jul 23 17:44:16.222780 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216402 2565 flags.go:64] FLAG: --storage-driver-user="root" Jul 23 17:44:16.222780 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216405 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 23 17:44:16.222780 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216408 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 23 17:44:16.222780 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216410 2565 flags.go:64] FLAG: --system-cgroups="" Jul 23 17:44:16.222780 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216413 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 23 17:44:16.222780 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216419 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 23 17:44:16.222780 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216422 2565 flags.go:64] FLAG: --tls-cert-file="" Jul 23 17:44:16.222780 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216425 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 23 17:44:16.222780 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216453 2565 flags.go:64] FLAG: --tls-min-version="" Jul 23 17:44:16.223373 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216457 2565 flags.go:64] FLAG: --tls-private-key-file="" Jul 23 17:44:16.223373 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216460 2565 flags.go:64] FLAG: --topology-manager-policy="none" Jul 23 17:44:16.223373 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216463 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 23 17:44:16.223373 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216466 2565 flags.go:64] FLAG: --topology-manager-scope="container" Jul 23 17:44:16.223373 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216469 2565 flags.go:64] FLAG: --v="2" Jul 23 17:44:16.223373 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216475 2565 flags.go:64] FLAG: --version="false" Jul 23 17:44:16.223373 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216479 2565 flags.go:64] FLAG: --vmodule="" Jul 23 17:44:16.223373 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216483 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 23 17:44:16.223373 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.216486 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 23 17:44:16.223373 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216620 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 17:44:16.223373 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216624 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 17:44:16.223373 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216627 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 17:44:16.223373 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216630 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 17:44:16.223373 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216634 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 17:44:16.223373 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216639 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 17:44:16.223373 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216642 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 17:44:16.223373 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216645 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 17:44:16.223373 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216649 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 17:44:16.223373 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216652 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 17:44:16.223373 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216655 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 17:44:16.223373 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216658 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 17:44:16.223954 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216660 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 17:44:16.223954 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216663 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 17:44:16.223954 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216665 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 17:44:16.223954 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216668 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 17:44:16.223954 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216670 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 17:44:16.223954 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216673 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 17:44:16.223954 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216675 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 17:44:16.223954 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216678 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 17:44:16.223954 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216680 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 17:44:16.223954 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216683 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 17:44:16.223954 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216685 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 17:44:16.223954 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216688 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 17:44:16.223954 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216690 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 17:44:16.223954 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216693 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 17:44:16.223954 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216695 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 17:44:16.223954 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216698 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 17:44:16.223954 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216702 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 17:44:16.223954 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216705 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 17:44:16.223954 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216708 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 17:44:16.223954 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216710 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 17:44:16.224430 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216713 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 17:44:16.224430 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216715 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 17:44:16.224430 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216718 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 17:44:16.224430 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216721 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 17:44:16.224430 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216723 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 17:44:16.224430 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216726 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 17:44:16.224430 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216729 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 17:44:16.224430 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216731 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 17:44:16.224430 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216734 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 17:44:16.224430 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216736 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 17:44:16.224430 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216739 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 17:44:16.224430 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216741 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 23 17:44:16.224430 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216744 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 17:44:16.224430 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216746 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 17:44:16.224430 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216748 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 17:44:16.224430 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216751 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 17:44:16.224430 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216753 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 17:44:16.224430 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216756 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 17:44:16.224430 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216760 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 17:44:16.224430 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216763 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 17:44:16.224922 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216766 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 17:44:16.224922 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216769 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 17:44:16.224922 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216772 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 17:44:16.224922 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216775 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 17:44:16.224922 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216778 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 17:44:16.224922 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216780 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 17:44:16.224922 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216783 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 17:44:16.224922 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216786 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 17:44:16.224922 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216789 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 17:44:16.224922 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216792 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 17:44:16.224922 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216795 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 17:44:16.224922 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216797 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 17:44:16.224922 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216800 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 17:44:16.224922 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216802 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 17:44:16.224922 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216805 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 17:44:16.224922 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216807 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 17:44:16.224922 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216810 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 17:44:16.224922 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216813 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 17:44:16.224922 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216815 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 17:44:16.224922 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216818 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 17:44:16.225388 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216820 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 17:44:16.225388 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216823 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 17:44:16.225388 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216825 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 17:44:16.225388 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216828 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 17:44:16.225388 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216830 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 17:44:16.225388 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216833 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 17:44:16.225388 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216835 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 17:44:16.225388 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216837 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 17:44:16.225388 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216840 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 17:44:16.225388 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216842 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 17:44:16.225388 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216845 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 17:44:16.225388 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216848 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 17:44:16.225388 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216850 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 17:44:16.225388 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216853 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 17:44:16.225388 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216855 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 17:44:16.225388 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216858 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 17:44:16.225388 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216860 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 17:44:16.225388 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216862 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 17:44:16.225388 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216865 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 17:44:16.225388 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216867 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 17:44:16.225891 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216872 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 17:44:16.225891 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216874 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 17:44:16.225891 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216877 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 17:44:16.225891 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216879 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 17:44:16.225891 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216882 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 17:44:16.225891 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216885 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 17:44:16.225891 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.216887 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 17:44:16.225891 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.217397 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 17:44:16.225891 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.224034 2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 23 17:44:16.225891 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.224050 2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 23 17:44:16.225891 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.224990 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 23 17:44:16.225891 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.224998 2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 23 17:44:16.225891 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225051 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 17:44:16.225891 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225056 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 17:44:16.225891 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225060 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 17:44:16.226263 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225063 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 17:44:16.226263 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225066 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 17:44:16.226263 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225069 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 17:44:16.226263 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225072 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 17:44:16.226263 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225075 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 17:44:16.226263 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225078 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 17:44:16.226263 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225081 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 17:44:16.226263 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225083 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 17:44:16.226263 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225086 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 17:44:16.226263 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225089 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 17:44:16.226263 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225092 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 17:44:16.226263 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225094 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 17:44:16.226263 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225097 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 17:44:16.226263 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225100 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 17:44:16.226263 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225102 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 17:44:16.226263 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225105 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 17:44:16.226263 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225108 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 17:44:16.226263 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225110 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 17:44:16.226263 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225113 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 17:44:16.226263 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225116 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 17:44:16.226801 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225118 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 17:44:16.226801 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225121 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 17:44:16.226801 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225123 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 17:44:16.226801 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225126 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 17:44:16.226801 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225128 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 17:44:16.226801 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225131 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 17:44:16.226801 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225135 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 17:44:16.226801 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225138 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 17:44:16.226801 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225140 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 17:44:16.226801 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225142 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 17:44:16.226801 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225145 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 17:44:16.226801 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225147 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 17:44:16.226801 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225150 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 17:44:16.226801 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225152 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 17:44:16.226801 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225154 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 17:44:16.226801 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225157 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 17:44:16.226801 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225159 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 17:44:16.226801 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225162 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 17:44:16.226801 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225165 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 17:44:16.226801 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225167 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 17:44:16.227279 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225169 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 17:44:16.227279 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225172 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 17:44:16.227279 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225174 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 17:44:16.227279 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225177 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 17:44:16.227279 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225179 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 17:44:16.227279 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225183 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 17:44:16.227279 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225185 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 17:44:16.227279 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225188 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 17:44:16.227279 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225190 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 17:44:16.227279 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225193 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 17:44:16.227279 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225196 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 17:44:16.227279 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225198 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 17:44:16.227279 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225201 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 17:44:16.227279 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225203 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 17:44:16.227279 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225206 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 17:44:16.227279 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225208 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 17:44:16.227279 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225211 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 17:44:16.227279 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225214 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 17:44:16.227279 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225217 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 17:44:16.227738 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225221 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 17:44:16.227738 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225225 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 17:44:16.227738 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225229 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 17:44:16.227738 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225232 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 17:44:16.227738 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225235 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 17:44:16.227738 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225237 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 17:44:16.227738 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225240 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 17:44:16.227738 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225243 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 17:44:16.227738 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225246 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 17:44:16.227738 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225248 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 17:44:16.227738 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225251 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 17:44:16.227738 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225255 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 17:44:16.227738 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225258 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 17:44:16.227738 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225261 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 17:44:16.227738 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225263 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 17:44:16.227738 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225266 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 17:44:16.227738 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225269 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 17:44:16.227738 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225271 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 17:44:16.227738 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225274 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 17:44:16.228229 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225276 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 17:44:16.228229 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225278 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 23 17:44:16.228229 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225281 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 17:44:16.228229 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225283 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 17:44:16.228229 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225286 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 17:44:16.228229 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225288 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 17:44:16.228229 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225291 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 17:44:16.228229 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225293 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 17:44:16.228229 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225296 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 17:44:16.228229 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225298 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 17:44:16.228229 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225301 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 17:44:16.228229 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225304 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 17:44:16.228229 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225308 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 17:44:16.228229 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225311 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 17:44:16.228229 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225315 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 17:44:16.228229 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225318 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 17:44:16.228229 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225320 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 17:44:16.228229 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225323 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 17:44:16.228665 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.225328 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 17:44:16.228665 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225433 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 17:44:16.228665 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225437 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 17:44:16.228665 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225439 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 17:44:16.228665 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225442 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 17:44:16.228665 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225444 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 17:44:16.228665 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225447 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 17:44:16.228665 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225449 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 17:44:16.228665 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225452 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 17:44:16.228665 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225454 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 17:44:16.228665 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225457 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 17:44:16.228665 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225460 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 17:44:16.228665 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225462 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 17:44:16.228665 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225464 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 17:44:16.228665 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225467 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 17:44:16.229024 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225470 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 17:44:16.229024 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225473 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 17:44:16.229024 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225477 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 17:44:16.229024 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225480 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 17:44:16.229024 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225483 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 17:44:16.229024 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225485 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 17:44:16.229024 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225488 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 17:44:16.229024 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225491 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 17:44:16.229024 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225493 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 17:44:16.229024 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225498 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 17:44:16.229024 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225501 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 17:44:16.229024 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225504 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 17:44:16.229024 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225507 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 17:44:16.229024 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225509 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 17:44:16.229024 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225513 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 17:44:16.229024 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225516 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 17:44:16.229024 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225518 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 17:44:16.229024 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225521 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 17:44:16.229024 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225523 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 17:44:16.229582 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225526 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 17:44:16.229582 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225529 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 17:44:16.229582 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225531 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 17:44:16.229582 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225534 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 17:44:16.229582 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225536 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 17:44:16.229582 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225539 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 17:44:16.229582 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225542 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 17:44:16.229582 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225546 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 17:44:16.229582 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225548 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 17:44:16.229582 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225551 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 17:44:16.229582 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225554 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 17:44:16.229582 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225556 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 17:44:16.229582 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225559 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 17:44:16.229582 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225561 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 17:44:16.229582 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225564 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 17:44:16.229582 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225566 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 17:44:16.229582 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225569 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 17:44:16.229582 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225571 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 17:44:16.229582 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225574 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 17:44:16.229582 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225576 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 17:44:16.230089 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225579 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 17:44:16.230089 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225581 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 17:44:16.230089 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225584 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 17:44:16.230089 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225586 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 17:44:16.230089 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225589 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 17:44:16.230089 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225591 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 17:44:16.230089 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225611 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 17:44:16.230089 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225614 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 17:44:16.230089 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225618 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 17:44:16.230089 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225620 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 17:44:16.230089 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225623 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 17:44:16.230089 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225625 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 17:44:16.230089 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225628 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 17:44:16.230089 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225630 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 17:44:16.230089 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225633 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 17:44:16.230089 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225635 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 17:44:16.230089 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225638 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 17:44:16.230089 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225640 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 17:44:16.230089 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225642 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 17:44:16.230760 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225645 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 17:44:16.230760 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225647 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 17:44:16.230760 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225650 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 17:44:16.230760 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225652 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 17:44:16.230760 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225655 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 17:44:16.230760 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225658 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 17:44:16.230760 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225661 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 17:44:16.230760 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225663 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 17:44:16.230760 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225666 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 17:44:16.230760 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225668 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 17:44:16.230760 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225672 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 17:44:16.230760 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225674 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 17:44:16.230760 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225677 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 17:44:16.230760 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225679 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 17:44:16.230760 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225682 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 17:44:16.230760 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225684 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 23 17:44:16.230760 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225687 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 17:44:16.230760 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225689 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 17:44:16.230760 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225692 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 17:44:16.230760 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225694 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 17:44:16.231467 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225697 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 17:44:16.231467 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225699 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 17:44:16.231467 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225702 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 17:44:16.231467 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225705 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 17:44:16.231467 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225707 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 17:44:16.231467 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225710 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 17:44:16.231467 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:16.225712 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 17:44:16.231467 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.225717 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 17:44:16.231467 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.225850 2565 server.go:962] "Client rotation is on, will bootstrap in background" Jul 23 17:44:16.231467 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.227845 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 23 17:44:16.231467 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.229200 2565 server.go:1019] "Starting client certificate rotation" Jul 23 17:44:16.231467 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.229319 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 17:44:16.231467 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.229371 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 17:44:16.232540 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.232524 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 17:44:16.233627 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.233608 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 17:44:16.245966 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.245949 2565 log.go:25] "Validated CRI v1 runtime API" Jul 23 17:44:16.248065 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.248051 2565 log.go:25] "Validated CRI v1 image API" Jul 23 17:44:16.248914 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.248895 2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 23 17:44:16.253555 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.253521 2565 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7fc2487c-02e0-4b6f-b672-011d7bb6af7e:/dev/nvme0n1p3 ed66ab95-957f-41f2-ba2f-556562eee6e1:/dev/nvme0n1p4] Jul 23 17:44:16.253654 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.253553 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 23 17:44:16.256046 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.256028 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 17:44:16.259400 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.259291 2565 manager.go:216] Machine: {Timestamp:2026-07-23 17:44:16.258153789 +0000 UTC m=+0.336457154 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100730 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec268c71c4ea2988d9b6262fe26eb877 SystemUUID:ec268c71-c4ea-2988-d9b6-262fe26eb877 BootID:312157c9-4a6f-4698-8b67-4e8b3bca25ba Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e9:96:7c:04:bd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e9:96:7c:04:bd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:36:b8:bf:0f:70 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 23 17:44:16.259400 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.259393 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 23 17:44:16.259531 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.259469 2565 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 23 17:44:16.263164 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.263139 2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 23 17:44:16.263294 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.263166 2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-82.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 23 17:44:16.263951 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.263941 2565 topology_manager.go:138] "Creating topology manager with none policy" Jul 23 17:44:16.263984 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.263954 2565 container_manager_linux.go:309] "Creating device plugin manager" Jul 23 17:44:16.263984 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.263961 2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 17:44:16.264853 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.264843 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 17:44:16.266248 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.266238 2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 23 17:44:16.269092 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.269081 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 23 17:44:16.269203 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.269193 2565 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 23 17:44:16.269242 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.269222 2565 kubelet.go:486] "Attempting to sync node with API server" Jul 23 17:44:16.269242 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.269232 2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 23 17:44:16.269304 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.269246 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 23 17:44:16.269304 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.269255 2565 kubelet.go:398] "Adding apiserver pod source" Jul 23 17:44:16.269304 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.269271 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 23 17:44:16.269674 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.269658 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 17:44:16.269725 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.269681 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 17:44:16.272477 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.272459 2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 23 17:44:16.274403 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.274388 2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 23 17:44:16.274461 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.274418 2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 23 17:44:16.275754 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.275738 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 23 17:44:16.275754 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.275755 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 23 17:44:16.275754 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.275765 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 23 17:44:16.275888 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.275772 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 23 17:44:16.275888 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.275780 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 23 17:44:16.275888 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.275786 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 23 17:44:16.275888 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.275794 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 23 17:44:16.275888 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.275800 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 23 17:44:16.275888 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.275807 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 23 17:44:16.275888 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.275813 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 23 17:44:16.275888 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.275821 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 23 17:44:16.275888 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.275827 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 23 17:44:16.277414 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.277403 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 23 17:44:16.277414 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.277413 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 23 17:44:16.279225 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:16.279203 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 23 17:44:16.279225 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:16.279203 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-82.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 23 17:44:16.279331 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.279270 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-82.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 23 17:44:16.280560 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.280546 2565 server.go:1268] "Started kubelet" Jul 23 17:44:16.280678 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.280656 2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 23 17:44:16.280678 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.280645 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 23 17:44:16.280824 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.280704 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 23 17:44:16.281303 ip-10-0-137-82 systemd[1]: Started Kubernetes Kubelet. Jul 23 17:44:16.282156 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.282129 2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 23 17:44:16.283365 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.283354 2565 server.go:310] "Adding debug handlers to kubelet server" Jul 23 17:44:16.289608 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:16.289578 2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 23 17:44:16.291954 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.291932 2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 23 17:44:16.291954 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.291948 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 23 17:44:16.292136 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:16.292095 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-82.ec2.internal\" not found" Jul 23 17:44:16.292971 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.292954 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 23 17:44:16.293070 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.292957 2565 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 23 17:44:16.293070 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.292988 2565 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 23 17:44:16.293070 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.293049 2565 factory.go:55] Registering systemd factory Jul 23 17:44:16.293211 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.293081 2565 reconstruct.go:97] "Volume reconstruction finished" Jul 23 17:44:16.293211 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.293090 2565 reconciler.go:29] "Reconciler: start to sync state" Jul 23 17:44:16.293211 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.293084 2565 factory.go:223] Registration of the systemd container factory successfully Jul 23 17:44:16.293327 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.293307 2565 factory.go:153] Registering CRI-O factory Jul 23 17:44:16.293327 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.293321 2565 factory.go:223] Registration of the crio container factory successfully Jul 23 17:44:16.293418 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.293373 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 23 17:44:16.293418 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.293397 2565 factory.go:103] Registering Raw factory Jul 23 17:44:16.293418 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.293411 2565 manager.go:1196] Started watching for new ooms in manager Jul 23 17:44:16.293824 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.293807 2565 manager.go:349] Starting recovery of all containers Jul 23 17:44:16.296643 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:16.296616 2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-82.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 23 17:44:16.296789 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:16.296768 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 23 17:44:16.298304 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:16.296618 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-82.ec2.internal.18c4fc34da875c0e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-82.ec2.internal,UID:ip-10-0-137-82.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-82.ec2.internal,},FirstTimestamp:2026-07-23 17:44:16.280361998 +0000 UTC m=+0.358665343,LastTimestamp:2026-07-23 17:44:16.280361998 +0000 UTC m=+0.358665343,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-82.ec2.internal,}" Jul 23 17:44:16.303796 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.303775 2565 manager.go:354] Recovery completed Jul 23 17:44:16.306117 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.306095 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pzswt" Jul 23 17:44:16.308268 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.308255 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 17:44:16.310205 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.310149 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-82.ec2.internal" event="NodeHasSufficientMemory" Jul 23 17:44:16.310205 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.310186 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-82.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 17:44:16.310205 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.310206 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-82.ec2.internal" event="NodeHasSufficientPID" Jul 23 17:44:16.310720 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.310708 2565 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 23 17:44:16.310720 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.310719 2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 23 17:44:16.310798 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.310728 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 17:44:16.310798 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.310734 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 23 17:44:16.311375 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.311359 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-82.ec2.internal" event="NodeHasSufficientMemory" Jul 23 17:44:16.311464 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.311387 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-82.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 17:44:16.311464 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.311402 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-82.ec2.internal" event="NodeHasSufficientPID" Jul 23 17:44:16.311547 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.311469 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pzswt" Jul 23 17:44:16.312053 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:16.311993 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-82.ec2.internal.18c4fc34dc4e4007 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-82.ec2.internal,UID:ip-10-0-137-82.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-82.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-82.ec2.internal,},FirstTimestamp:2026-07-23 17:44:16.310173703 +0000 UTC m=+0.388477054,LastTimestamp:2026-07-23 17:44:16.310173703 +0000 UTC m=+0.388477054,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-82.ec2.internal,}" Jul 23 17:44:16.312871 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.312859 2565 policy_none.go:49] "None policy: Start" Jul 23 17:44:16.312927 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.312874 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 23 17:44:16.312927 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.312886 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 23 17:44:16.313925 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.313915 2565 policy_none.go:47] "Start" Jul 23 17:44:16.346357 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.346202 2565 manager.go:337] "Starting Device Plugin manager" Jul 23 17:44:16.358007 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:16.346376 2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 23 17:44:16.358007 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.346390 2565 server.go:87] "Starting device plugin registration server" Jul 23 17:44:16.358007 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.346633 2565 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 23 17:44:16.358007 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.346643 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 23 17:44:16.358007 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.346753 2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 23 17:44:16.358007 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.346838 2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 23 17:44:16.358007 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.346847 2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 23 17:44:16.358007 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:16.347310 2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 23 17:44:16.358007 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:16.347363 2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-82.ec2.internal\" not found" Jul 23 17:44:16.398026 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.397997 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 23 17:44:16.399170 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.399155 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 23 17:44:16.399279 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.399192 2565 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 23 17:44:16.399279 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.399214 2565 kubelet.go:2448] "Starting kubelet main sync loop" Jul 23 17:44:16.399279 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:16.399246 2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 23 17:44:16.401672 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.401652 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 17:44:16.447372 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.447345 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 17:44:16.448218 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.448198 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-82.ec2.internal" event="NodeHasSufficientMemory" Jul 23 17:44:16.448322 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.448230 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-82.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 17:44:16.448322 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.448246 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-82.ec2.internal" event="NodeHasSufficientPID" Jul 23 17:44:16.448322 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.448276 2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-82.ec2.internal" Jul 23 17:44:16.457546 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.457530 2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-82.ec2.internal" Jul 23 17:44:16.457610 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:16.457550 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-82.ec2.internal\": node \"ip-10-0-137-82.ec2.internal\" not found" Jul 23 17:44:16.483405 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:16.483387 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-82.ec2.internal\" not found" Jul 23 17:44:16.499850 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.499827 2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-82.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-82.ec2.internal"] Jul 23 17:44:16.499949 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.499888 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 17:44:16.501420 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.501403 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-82.ec2.internal" event="NodeHasSufficientMemory" Jul 23 17:44:16.501502 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.501438 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-82.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 17:44:16.501502 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.501454 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-82.ec2.internal" event="NodeHasSufficientPID" Jul 23 17:44:16.502699 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.502685 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 17:44:16.502865 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.502852 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-82.ec2.internal" Jul 23 17:44:16.502914 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.502883 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 17:44:16.503374 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.503358 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-82.ec2.internal" event="NodeHasSufficientMemory" Jul 23 17:44:16.503420 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.503387 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-82.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 17:44:16.503420 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.503398 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-82.ec2.internal" event="NodeHasSufficientPID" Jul 23 17:44:16.503489 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.503358 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-82.ec2.internal" event="NodeHasSufficientMemory" Jul 23 17:44:16.503489 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.503458 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-82.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 17:44:16.503489 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.503471 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-82.ec2.internal" event="NodeHasSufficientPID" Jul 23 17:44:16.504632 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.504608 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-82.ec2.internal" Jul 23 17:44:16.504718 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.504643 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 17:44:16.505282 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.505264 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-82.ec2.internal" event="NodeHasSufficientMemory" Jul 23 17:44:16.505361 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.505291 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-82.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 17:44:16.505361 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.505304 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-82.ec2.internal" event="NodeHasSufficientPID" Jul 23 17:44:16.520296 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:16.520274 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-82.ec2.internal\" not found" node="ip-10-0-137-82.ec2.internal" Jul 23 17:44:16.524195 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:16.524180 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-82.ec2.internal\" not found" node="ip-10-0-137-82.ec2.internal" Jul 23 17:44:16.584328 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:16.584243 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-82.ec2.internal\" not found" Jul 23 17:44:16.595699 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.595677 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4caf25fc2a9f5a4108b00096ab025031-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-82.ec2.internal\" (UID: \"4caf25fc2a9f5a4108b00096ab025031\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-82.ec2.internal" Jul 23 17:44:16.595823 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.595704 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4caf25fc2a9f5a4108b00096ab025031-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-82.ec2.internal\" (UID: \"4caf25fc2a9f5a4108b00096ab025031\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-82.ec2.internal" Jul 23 17:44:16.595823 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.595722 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/175ceae0d598475e91b574d725c1c32d-config\") pod \"kube-apiserver-proxy-ip-10-0-137-82.ec2.internal\" (UID: \"175ceae0d598475e91b574d725c1c32d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-82.ec2.internal" Jul 23 17:44:16.684810 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:16.684783 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-82.ec2.internal\" not found" Jul 23 17:44:16.696250 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.696230 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4caf25fc2a9f5a4108b00096ab025031-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-82.ec2.internal\" (UID: \"4caf25fc2a9f5a4108b00096ab025031\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-82.ec2.internal" Jul 23 17:44:16.696305 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.696175 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4caf25fc2a9f5a4108b00096ab025031-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-82.ec2.internal\" (UID: \"4caf25fc2a9f5a4108b00096ab025031\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-82.ec2.internal" Jul 23 17:44:16.696305 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.696298 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4caf25fc2a9f5a4108b00096ab025031-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-82.ec2.internal\" (UID: \"4caf25fc2a9f5a4108b00096ab025031\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-82.ec2.internal" Jul 23 17:44:16.696403 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.696316 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/175ceae0d598475e91b574d725c1c32d-config\") pod \"kube-apiserver-proxy-ip-10-0-137-82.ec2.internal\" (UID: \"175ceae0d598475e91b574d725c1c32d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-82.ec2.internal" Jul 23 17:44:16.696403 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.696379 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4caf25fc2a9f5a4108b00096ab025031-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-82.ec2.internal\" (UID: \"4caf25fc2a9f5a4108b00096ab025031\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-82.ec2.internal" Jul 23 17:44:16.696471 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.696421 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/175ceae0d598475e91b574d725c1c32d-config\") pod \"kube-apiserver-proxy-ip-10-0-137-82.ec2.internal\" (UID: \"175ceae0d598475e91b574d725c1c32d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-82.ec2.internal" Jul 23 17:44:16.785674 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:16.785637 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-82.ec2.internal\" not found" Jul 23 17:44:16.824195 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.824162 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-82.ec2.internal" Jul 23 17:44:16.826323 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:16.826305 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-82.ec2.internal" Jul 23 17:44:16.886476 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:16.886407 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-82.ec2.internal\" not found" Jul 23 17:44:16.986991 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:16.986958 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-82.ec2.internal\" not found" Jul 23 17:44:17.090725 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:17.090702 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-82.ec2.internal\" not found" Jul 23 17:44:17.194000 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:17.193924 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-82.ec2.internal\" not found" Jul 23 17:44:17.229245 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:17.229215 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 23 17:44:17.229728 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:17.229368 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 17:44:17.295039 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:17.295007 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-82.ec2.internal\" not found" Jul 23 17:44:17.316205 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:17.316155 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 17:39:16 +0000 UTC" deadline="2028-02-01 19:39:28.698447153 +0000 UTC" Jul 23 17:44:17.316205 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:17.316203 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13393h55m11.382248232s" Jul 23 17:44:17.395943 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:17.395915 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-82.ec2.internal\" not found" Jul 23 17:44:17.430817 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:17.430790 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 17:44:17.496743 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:17.496720 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-82.ec2.internal\" not found" Jul 23 17:44:17.501235 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:17.501203 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4caf25fc2a9f5a4108b00096ab025031.slice/crio-70c701b6c304451bc6d849c730d4e14f418a0ae37da9affec529f7a3c713a72e WatchSource:0}: Error finding container 70c701b6c304451bc6d849c730d4e14f418a0ae37da9affec529f7a3c713a72e: Status 404 returned error can't find the container with id 70c701b6c304451bc6d849c730d4e14f418a0ae37da9affec529f7a3c713a72e Jul 23 17:44:17.501736 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:17.501708 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175ceae0d598475e91b574d725c1c32d.slice/crio-559445c4d524f2cb2bfdd7934907e656a619a733df2a1187f2ca10836efe58b7 WatchSource:0}: Error finding container 559445c4d524f2cb2bfdd7934907e656a619a733df2a1187f2ca10836efe58b7: Status 404 returned error can't find the container with id 559445c4d524f2cb2bfdd7934907e656a619a733df2a1187f2ca10836efe58b7 Jul 23 17:44:17.507162 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:17.507142 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 17:44:17.597726 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:17.597687 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-82.ec2.internal\" not found" Jul 23 17:44:17.691656 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:17.691631 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 17:44:17.697925 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:17.697902 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-82.ec2.internal\" not found" Jul 23 17:44:17.798105 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:17.798034 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-82.ec2.internal\" not found" Jul 23 17:44:17.832410 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:17.832384 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 17:44:17.894051 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:17.894023 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-82.ec2.internal" Jul 23 17:44:17.910673 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:17.910647 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 17:44:17.911995 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:17.911978 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-82.ec2.internal" Jul 23 17:44:17.922279 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:17.922257 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 17:44:18.271087 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.270762 2565 apiserver.go:52] "Watching apiserver" Jul 23 17:44:18.282834 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.282801 2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 23 17:44:18.283761 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.283731 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-vcfvv","openshift-multus/network-metrics-daemon-t2hlf","openshift-network-diagnostics/network-check-target-slcq6","openshift-ovn-kubernetes/ovnkube-node-pnj76","kube-system/konnectivity-agent-kcd4h","kube-system/kube-apiserver-proxy-ip-10-0-137-82.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk8n6","openshift-image-registry/node-ca-m8496","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-82.ec2.internal","openshift-multus/multus-ggqsr","openshift-network-operator/iptables-alerter-rtmx5","kube-system/global-pull-secret-syncer-2j6vd","openshift-cluster-node-tuning-operator/tuned-5xjz4","openshift-dns/node-resolver-6nsmg"] Jul 23 17:44:18.285801 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.285776 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk8n6" Jul 23 17:44:18.287111 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.287088 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.288073 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.288053 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slcq6" Jul 23 17:44:18.288172 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:18.288142 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slcq6" podUID="cea169f8-ef99-4ce1-9559-029a6507402f" Jul 23 17:44:18.288532 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.288510 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bqp2q\"" Jul 23 17:44:18.288857 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.288839 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 23 17:44:18.289073 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.289054 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 23 17:44:18.289232 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.289196 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 23 17:44:18.289412 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.289393 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 23 17:44:18.289590 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.289569 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f642k\"" Jul 23 17:44:18.289650 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.289638 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 23 17:44:18.291010 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.290986 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.292937 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.292917 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 23 17:44:18.293422 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.293402 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 23 17:44:18.293817 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.293796 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 23 17:44:18.293969 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.293949 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 23 17:44:18.294121 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.294101 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 23 17:44:18.294593 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.294572 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kcd4h" Jul 23 17:44:18.295399 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.295379 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 23 17:44:18.295614 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.295592 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kk7cv\"" Jul 23 17:44:18.295826 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.295807 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 23 17:44:18.296182 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.296165 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6nsmg" Jul 23 17:44:18.297469 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.297452 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m8496" Jul 23 17:44:18.299257 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.298886 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-kzq6q\"" Jul 23 17:44:18.299257 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.298989 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 23 17:44:18.299257 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.299114 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 23 17:44:18.299450 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.299316 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vcfvv" Jul 23 17:44:18.299580 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.299539 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5zsgj\"" Jul 23 17:44:18.300482 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.300032 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 23 17:44:18.300482 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.300224 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 23 17:44:18.300674 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.300647 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:44:18.300741 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:18.300721 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2hlf" podUID="d7027def-1a58-4d31-a9ce-ea9814a550a3" Jul 23 17:44:18.301535 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.301018 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 23 17:44:18.301535 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.301028 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 23 17:44:18.301535 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.301149 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 23 17:44:18.301535 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.301303 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mcrkd\"" Jul 23 17:44:18.302498 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.302091 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 23 17:44:18.302498 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.302301 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 23 17:44:18.302498 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.302314 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 23 17:44:18.302498 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.302399 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tc9gt\"" Jul 23 17:44:18.302762 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.302700 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 23 17:44:18.302970 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.302948 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.303449 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.303173 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 23 17:44:18.303803 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.303782 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rtmx5" Jul 23 17:44:18.305108 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.305082 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-systemd-units\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.305201 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.305122 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-host-run-netns\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.305201 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.305149 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-var-lib-openvswitch\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.305201 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.305173 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-etc-openvswitch\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.305201 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.305199 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msflt\" (UniqueName: \"kubernetes.io/projected/84ce7a4f-1e92-434b-b939-616cce7d9188-kube-api-access-msflt\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.305408 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.305226 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4a324aac-61f5-46a9-b847-15730ba25434-konnectivity-ca\") pod \"konnectivity-agent-kcd4h\" (UID: \"4a324aac-61f5-46a9-b847-15730ba25434\") " pod="kube-system/konnectivity-agent-kcd4h" Jul 23 17:44:18.305408 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.305253 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-host-slash\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.305408 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.305277 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-node-log\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.305408 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.305303 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98b7a47e-3e44-401b-8d69-0052ef88824d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qk8n6\" (UID: \"98b7a47e-3e44-401b-8d69-0052ef88824d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk8n6" Jul 23 17:44:18.305408 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.305326 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-etc-modprobe-d\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.305408 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.305350 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-var-lib-kubelet\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.305408 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.305373 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjsc5\" (UniqueName: \"kubernetes.io/projected/bcc857f9-c104-4971-a960-86f234b26a90-kube-api-access-sjsc5\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.305759 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.305420 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.305759 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.305430 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:18.305759 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:18.305511 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2j6vd" podUID="39d55095-246a-4ecf-bb04-36a9a8c92da4" Jul 23 17:44:18.305759 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.305687 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 23 17:44:18.305963 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.305773 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 17:44:18.305963 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.305840 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84ce7a4f-1e92-434b-b939-616cce7d9188-env-overrides\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.305963 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.305884 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98b7a47e-3e44-401b-8d69-0052ef88824d-socket-dir\") pod \"aws-ebs-csi-driver-node-qk8n6\" (UID: \"98b7a47e-3e44-401b-8d69-0052ef88824d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk8n6" Jul 23 17:44:18.305963 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.305929 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-sys\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.306161 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.305976 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-host-kubelet\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.306161 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.305999 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-run-openvswitch\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.306161 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.306014 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-96q58\"" Jul 23 17:44:18.306161 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.306036 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-run-ovn\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.306351 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.306061 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-etc-kubernetes\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.306406 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.306380 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-etc-sysctl-d\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.306458 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.306407 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-run\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.306458 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.306432 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcc857f9-c104-4971-a960-86f234b26a90-tmp\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.306458 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.306452 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 23 17:44:18.306605 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.306457 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-host-run-ovn-kubernetes\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.306605 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.306481 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84ce7a4f-1e92-434b-b939-616cce7d9188-ovnkube-config\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.306605 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.306489 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 23 17:44:18.306605 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.306505 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84ce7a4f-1e92-434b-b939-616cce7d9188-ovn-node-metrics-cert\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.306605 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.306532 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98b7a47e-3e44-401b-8d69-0052ef88824d-registration-dir\") pod \"aws-ebs-csi-driver-node-qk8n6\" (UID: \"98b7a47e-3e44-401b-8d69-0052ef88824d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk8n6" Jul 23 17:44:18.306605 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.306557 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98b7a47e-3e44-401b-8d69-0052ef88824d-etc-selinux\") pod \"aws-ebs-csi-driver-node-qk8n6\" (UID: \"98b7a47e-3e44-401b-8d69-0052ef88824d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk8n6" Jul 23 17:44:18.306605 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.306600 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sszl\" (UniqueName: \"kubernetes.io/projected/98b7a47e-3e44-401b-8d69-0052ef88824d-kube-api-access-6sszl\") pod \"aws-ebs-csi-driver-node-qk8n6\" (UID: \"98b7a47e-3e44-401b-8d69-0052ef88824d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk8n6" Jul 23 17:44:18.306947 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.306624 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-etc-sysctl-conf\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.306947 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.306663 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-lib-modules\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.306947 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.306687 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lsk7x\"" Jul 23 17:44:18.306947 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.306738 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4a324aac-61f5-46a9-b847-15730ba25434-agent-certs\") pod \"konnectivity-agent-kcd4h\" (UID: \"4a324aac-61f5-46a9-b847-15730ba25434\") " pod="kube-system/konnectivity-agent-kcd4h" Jul 23 17:44:18.306947 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.306769 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98b7a47e-3e44-401b-8d69-0052ef88824d-device-dir\") pod \"aws-ebs-csi-driver-node-qk8n6\" (UID: \"98b7a47e-3e44-401b-8d69-0052ef88824d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk8n6" Jul 23 17:44:18.306947 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.306787 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 23 17:44:18.306947 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.306794 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqwqs\" (UniqueName: \"kubernetes.io/projected/cea169f8-ef99-4ce1-9559-029a6507402f-kube-api-access-dqwqs\") pod \"network-check-target-slcq6\" (UID: \"cea169f8-ef99-4ce1-9559-029a6507402f\") " pod="openshift-network-diagnostics/network-check-target-slcq6" Jul 23 17:44:18.306947 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.306851 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-run-systemd\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.306947 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.306878 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-log-socket\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.306947 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.306921 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-host-cni-bin\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.307400 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.307043 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-host-cni-netd\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.307400 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.307081 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98b7a47e-3e44-401b-8d69-0052ef88824d-sys-fs\") pod \"aws-ebs-csi-driver-node-qk8n6\" (UID: \"98b7a47e-3e44-401b-8d69-0052ef88824d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk8n6" Jul 23 17:44:18.307400 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.307105 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-host\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.307400 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.307129 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84ce7a4f-1e92-434b-b939-616cce7d9188-ovnkube-script-lib\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.307400 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.307153 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-etc-sysconfig\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.307400 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.307175 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-etc-systemd\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.307400 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.307197 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bcc857f9-c104-4971-a960-86f234b26a90-etc-tuned\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.328472 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.328451 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fk825" Jul 23 17:44:18.337655 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.337294 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fk825" Jul 23 17:44:18.394701 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.394625 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 23 17:44:18.404400 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.404343 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-82.ec2.internal" event={"ID":"175ceae0d598475e91b574d725c1c32d","Type":"ContainerStarted","Data":"559445c4d524f2cb2bfdd7934907e656a619a733df2a1187f2ca10836efe58b7"} Jul 23 17:44:18.405743 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.405697 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-82.ec2.internal" event={"ID":"4caf25fc2a9f5a4108b00096ab025031","Type":"ContainerStarted","Data":"70c701b6c304451bc6d849c730d4e14f418a0ae37da9affec529f7a3c713a72e"} Jul 23 17:44:18.407966 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.407946 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-cnibin\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.408069 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.407989 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6b9t\" (UniqueName: \"kubernetes.io/projected/91f8ae79-5ace-4707-82f0-c5a5770eba19-kube-api-access-m6b9t\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.408069 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408017 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1112f49-7d09-4c67-b66d-29e03194d1f9-host-slash\") pod \"iptables-alerter-rtmx5\" (UID: \"b1112f49-7d09-4c67-b66d-29e03194d1f9\") " pod="openshift-network-operator/iptables-alerter-rtmx5" Jul 23 17:44:18.408069 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408057 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-host-run-netns\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.408227 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408081 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-var-lib-openvswitch\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.408227 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408107 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4a324aac-61f5-46a9-b847-15730ba25434-konnectivity-ca\") pod \"konnectivity-agent-kcd4h\" (UID: \"4a324aac-61f5-46a9-b847-15730ba25434\") " pod="kube-system/konnectivity-agent-kcd4h" Jul 23 17:44:18.408227 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408130 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-multus-cni-dir\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.408227 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408156 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-host-var-lib-cni-multus\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.408227 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408179 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce-cnibin\") pod \"multus-additional-cni-plugins-vcfvv\" (UID: \"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvv" Jul 23 17:44:18.408227 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408201 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vcfvv\" (UID: \"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvv" Jul 23 17:44:18.408227 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408226 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-host-var-lib-kubelet\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.408556 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408249 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-multus-conf-dir\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.408556 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408272 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39d55095-246a-4ecf-bb04-36a9a8c92da4-original-pull-secret\") pod \"global-pull-secret-syncer-2j6vd\" (UID: \"39d55095-246a-4ecf-bb04-36a9a8c92da4\") " pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:18.408556 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408298 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.408556 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408325 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-sys\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.408556 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408349 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b11ec86f-5925-4385-aeca-5b5e88f693fa-serviceca\") pod \"node-ca-m8496\" (UID: \"b11ec86f-5925-4385-aeca-5b5e88f693fa\") " pod="openshift-image-registry/node-ca-m8496" Jul 23 17:44:18.408556 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408375 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcmjq\" (UniqueName: \"kubernetes.io/projected/b11ec86f-5925-4385-aeca-5b5e88f693fa-kube-api-access-rcmjq\") pod \"node-ca-m8496\" (UID: \"b11ec86f-5925-4385-aeca-5b5e88f693fa\") " pod="openshift-image-registry/node-ca-m8496" Jul 23 17:44:18.408556 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408437 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.408556 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408475 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-sys\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.408556 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408485 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-var-lib-openvswitch\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.408556 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408440 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vcfvv\" (UID: \"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvv" Jul 23 17:44:18.408556 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408522 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-host-run-netns\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.409099 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408568 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69063b0f-0422-463f-833c-5aecf8d331a7-tmp-dir\") pod \"node-resolver-6nsmg\" (UID: \"69063b0f-0422-463f-833c-5aecf8d331a7\") " pod="openshift-dns/node-resolver-6nsmg" Jul 23 17:44:18.409099 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408671 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-host-kubelet\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.409099 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408698 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-run-openvswitch\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.409099 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408722 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-run-ovn\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.409099 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408746 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-etc-sysctl-d\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.409099 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408774 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-run-openvswitch\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.409099 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408787 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-host-run-ovn-kubernetes\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.409099 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408794 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-run-ovn\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.409099 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408757 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-host-kubelet\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.409099 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408809 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84ce7a4f-1e92-434b-b939-616cce7d9188-ovn-node-metrics-cert\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.409099 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408836 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98b7a47e-3e44-401b-8d69-0052ef88824d-etc-selinux\") pod \"aws-ebs-csi-driver-node-qk8n6\" (UID: \"98b7a47e-3e44-401b-8d69-0052ef88824d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk8n6" Jul 23 17:44:18.409099 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408844 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-host-run-ovn-kubernetes\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.409099 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408854 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-lib-modules\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.409099 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408874 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/39d55095-246a-4ecf-bb04-36a9a8c92da4-dbus\") pod \"global-pull-secret-syncer-2j6vd\" (UID: \"39d55095-246a-4ecf-bb04-36a9a8c92da4\") " pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:18.409099 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408897 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98b7a47e-3e44-401b-8d69-0052ef88824d-device-dir\") pod \"aws-ebs-csi-driver-node-qk8n6\" (UID: \"98b7a47e-3e44-401b-8d69-0052ef88824d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk8n6" Jul 23 17:44:18.409099 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408920 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-hostroot\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.409099 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408960 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-etc-sysctl-d\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.409911 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408972 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/91f8ae79-5ace-4707-82f0-c5a5770eba19-multus-daemon-config\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.409911 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.408997 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-host-run-multus-certs\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.409911 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409018 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce-os-release\") pod \"multus-additional-cni-plugins-vcfvv\" (UID: \"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvv" Jul 23 17:44:18.409911 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409042 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-log-socket\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.409911 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409057 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98b7a47e-3e44-401b-8d69-0052ef88824d-etc-selinux\") pod \"aws-ebs-csi-driver-node-qk8n6\" (UID: \"98b7a47e-3e44-401b-8d69-0052ef88824d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk8n6" Jul 23 17:44:18.409911 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409067 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-host-cni-bin\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.409911 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409090 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-host-cni-netd\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.409911 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409108 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4a324aac-61f5-46a9-b847-15730ba25434-konnectivity-ca\") pod \"konnectivity-agent-kcd4h\" (UID: \"4a324aac-61f5-46a9-b847-15730ba25434\") " pod="kube-system/konnectivity-agent-kcd4h" Jul 23 17:44:18.409911 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409112 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98b7a47e-3e44-401b-8d69-0052ef88824d-sys-fs\") pod \"aws-ebs-csi-driver-node-qk8n6\" (UID: \"98b7a47e-3e44-401b-8d69-0052ef88824d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk8n6" Jul 23 17:44:18.409911 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409153 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-lib-modules\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.409911 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409158 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98b7a47e-3e44-401b-8d69-0052ef88824d-sys-fs\") pod \"aws-ebs-csi-driver-node-qk8n6\" (UID: \"98b7a47e-3e44-401b-8d69-0052ef88824d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk8n6" Jul 23 17:44:18.409911 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409154 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-host\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.409911 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409168 2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 23 17:44:18.409911 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409206 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-host\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.409911 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409195 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-host-run-netns\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.409911 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409251 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98b7a47e-3e44-401b-8d69-0052ef88824d-device-dir\") pod \"aws-ebs-csi-driver-node-qk8n6\" (UID: \"98b7a47e-3e44-401b-8d69-0052ef88824d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk8n6" Jul 23 17:44:18.409911 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409299 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-log-socket\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.409911 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409329 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-host-cni-netd\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.410746 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409333 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-host-cni-bin\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.410746 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409343 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbm6x\" (UniqueName: \"kubernetes.io/projected/5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce-kube-api-access-nbm6x\") pod \"multus-additional-cni-plugins-vcfvv\" (UID: \"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvv" Jul 23 17:44:18.410746 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409394 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-etc-sysconfig\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.410746 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409429 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/69063b0f-0422-463f-833c-5aecf8d331a7-hosts-file\") pod \"node-resolver-6nsmg\" (UID: \"69063b0f-0422-463f-833c-5aecf8d331a7\") " pod="openshift-dns/node-resolver-6nsmg" Jul 23 17:44:18.410746 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409459 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-etc-sysconfig\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.410746 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409494 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91f8ae79-5ace-4707-82f0-c5a5770eba19-cni-binary-copy\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.410746 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409520 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-host-run-k8s-cni-cncf-io\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.410746 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409566 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce-system-cni-dir\") pod \"multus-additional-cni-plugins-vcfvv\" (UID: \"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvv" Jul 23 17:44:18.410746 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409617 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-systemd-units\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.410746 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409645 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-etc-openvswitch\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.410746 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409683 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msflt\" (UniqueName: \"kubernetes.io/projected/84ce7a4f-1e92-434b-b939-616cce7d9188-kube-api-access-msflt\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.410746 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409710 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vcfvv\" (UID: \"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvv" Jul 23 17:44:18.410746 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409736 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-host-slash\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.410746 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409761 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-node-log\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.410746 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409784 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98b7a47e-3e44-401b-8d69-0052ef88824d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qk8n6\" (UID: \"98b7a47e-3e44-401b-8d69-0052ef88824d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk8n6" Jul 23 17:44:18.410746 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409808 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-etc-modprobe-d\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.410746 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409860 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-var-lib-kubelet\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.411573 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409884 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-system-cni-dir\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.411573 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409906 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-etc-kubernetes\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.411573 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409945 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/39d55095-246a-4ecf-bb04-36a9a8c92da4-kubelet-config\") pod \"global-pull-secret-syncer-2j6vd\" (UID: \"39d55095-246a-4ecf-bb04-36a9a8c92da4\") " pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:18.411573 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409972 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84ce7a4f-1e92-434b-b939-616cce7d9188-env-overrides\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.411573 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.409997 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98b7a47e-3e44-401b-8d69-0052ef88824d-socket-dir\") pod \"aws-ebs-csi-driver-node-qk8n6\" (UID: \"98b7a47e-3e44-401b-8d69-0052ef88824d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk8n6" Jul 23 17:44:18.411573 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.410018 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b11ec86f-5925-4385-aeca-5b5e88f693fa-host\") pod \"node-ca-m8496\" (UID: \"b11ec86f-5925-4385-aeca-5b5e88f693fa\") " pod="openshift-image-registry/node-ca-m8496" Jul 23 17:44:18.411573 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.410041 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjnvl\" (UniqueName: \"kubernetes.io/projected/d7027def-1a58-4d31-a9ce-ea9814a550a3-kube-api-access-fjnvl\") pod \"network-metrics-daemon-t2hlf\" (UID: \"d7027def-1a58-4d31-a9ce-ea9814a550a3\") " pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:44:18.411573 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.410066 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rvs4\" (UniqueName: \"kubernetes.io/projected/b1112f49-7d09-4c67-b66d-29e03194d1f9-kube-api-access-8rvs4\") pod \"iptables-alerter-rtmx5\" (UID: \"b1112f49-7d09-4c67-b66d-29e03194d1f9\") " pod="openshift-network-operator/iptables-alerter-rtmx5" Jul 23 17:44:18.411573 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.410092 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjsc5\" (UniqueName: \"kubernetes.io/projected/bcc857f9-c104-4971-a960-86f234b26a90-kube-api-access-sjsc5\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.411573 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.410118 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-etc-kubernetes\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.411573 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.410154 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-run\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.411573 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.410250 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-etc-kubernetes\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.411573 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.410280 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcc857f9-c104-4971-a960-86f234b26a90-tmp\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.411573 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.410309 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce-cni-binary-copy\") pod \"multus-additional-cni-plugins-vcfvv\" (UID: \"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvv" Jul 23 17:44:18.411573 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.410335 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7027def-1a58-4d31-a9ce-ea9814a550a3-metrics-certs\") pod \"network-metrics-daemon-t2hlf\" (UID: \"d7027def-1a58-4d31-a9ce-ea9814a550a3\") " pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:44:18.411573 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.410350 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98b7a47e-3e44-401b-8d69-0052ef88824d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qk8n6\" (UID: \"98b7a47e-3e44-401b-8d69-0052ef88824d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk8n6" Jul 23 17:44:18.411573 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.410362 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84ce7a4f-1e92-434b-b939-616cce7d9188-ovnkube-config\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.412461 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.410389 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98b7a47e-3e44-401b-8d69-0052ef88824d-registration-dir\") pod \"aws-ebs-csi-driver-node-qk8n6\" (UID: \"98b7a47e-3e44-401b-8d69-0052ef88824d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk8n6" Jul 23 17:44:18.412461 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.410438 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sszl\" (UniqueName: \"kubernetes.io/projected/98b7a47e-3e44-401b-8d69-0052ef88824d-kube-api-access-6sszl\") pod \"aws-ebs-csi-driver-node-qk8n6\" (UID: \"98b7a47e-3e44-401b-8d69-0052ef88824d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk8n6" Jul 23 17:44:18.412461 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.410456 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-etc-openvswitch\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.412461 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.410466 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-etc-sysctl-conf\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.412461 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.410492 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-os-release\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.412461 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.410486 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-systemd-units\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.412461 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.410505 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-var-lib-kubelet\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.412461 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.410459 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-run\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.412461 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.410549 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-host-var-lib-cni-bin\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.412461 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.410582 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98b7a47e-3e44-401b-8d69-0052ef88824d-registration-dir\") pod \"aws-ebs-csi-driver-node-qk8n6\" (UID: \"98b7a47e-3e44-401b-8d69-0052ef88824d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk8n6" Jul 23 17:44:18.412461 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.410768 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b1112f49-7d09-4c67-b66d-29e03194d1f9-iptables-alerter-script\") pod \"iptables-alerter-rtmx5\" (UID: \"b1112f49-7d09-4c67-b66d-29e03194d1f9\") " pod="openshift-network-operator/iptables-alerter-rtmx5" Jul 23 17:44:18.412461 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.410778 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-etc-modprobe-d\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.412461 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.410806 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4a324aac-61f5-46a9-b847-15730ba25434-agent-certs\") pod \"konnectivity-agent-kcd4h\" (UID: \"4a324aac-61f5-46a9-b847-15730ba25434\") " pod="kube-system/konnectivity-agent-kcd4h" Jul 23 17:44:18.412461 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.410839 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-multus-socket-dir-parent\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.412461 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.410907 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-host-slash\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.412461 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.410968 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-node-log\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.412461 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.411095 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-etc-sysctl-conf\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.413274 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.411104 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84ce7a4f-1e92-434b-b939-616cce7d9188-env-overrides\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.413274 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.411203 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98b7a47e-3e44-401b-8d69-0052ef88824d-socket-dir\") pod \"aws-ebs-csi-driver-node-qk8n6\" (UID: \"98b7a47e-3e44-401b-8d69-0052ef88824d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk8n6" Jul 23 17:44:18.413274 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.411302 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqwqs\" (UniqueName: \"kubernetes.io/projected/cea169f8-ef99-4ce1-9559-029a6507402f-kube-api-access-dqwqs\") pod \"network-check-target-slcq6\" (UID: \"cea169f8-ef99-4ce1-9559-029a6507402f\") " pod="openshift-network-diagnostics/network-check-target-slcq6" Jul 23 17:44:18.413274 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.411310 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84ce7a4f-1e92-434b-b939-616cce7d9188-ovnkube-config\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.413274 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.411331 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-run-systemd\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.413274 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.411356 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bcc857f9-c104-4971-a960-86f234b26a90-etc-tuned\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.413274 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.411383 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2z7m\" (UniqueName: \"kubernetes.io/projected/69063b0f-0422-463f-833c-5aecf8d331a7-kube-api-access-k2z7m\") pod \"node-resolver-6nsmg\" (UID: \"69063b0f-0422-463f-833c-5aecf8d331a7\") " pod="openshift-dns/node-resolver-6nsmg" Jul 23 17:44:18.413274 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.411393 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84ce7a4f-1e92-434b-b939-616cce7d9188-run-systemd\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.413274 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.411409 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84ce7a4f-1e92-434b-b939-616cce7d9188-ovnkube-script-lib\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.413274 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.411433 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-etc-systemd\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.413274 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.411555 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bcc857f9-c104-4971-a960-86f234b26a90-etc-systemd\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.413274 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.412098 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84ce7a4f-1e92-434b-b939-616cce7d9188-ovnkube-script-lib\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.413274 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.413097 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84ce7a4f-1e92-434b-b939-616cce7d9188-ovn-node-metrics-cert\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.414587 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.414565 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4a324aac-61f5-46a9-b847-15730ba25434-agent-certs\") pod \"konnectivity-agent-kcd4h\" (UID: \"4a324aac-61f5-46a9-b847-15730ba25434\") " pod="kube-system/konnectivity-agent-kcd4h" Jul 23 17:44:18.414906 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.414884 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bcc857f9-c104-4971-a960-86f234b26a90-etc-tuned\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.415996 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.415959 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcc857f9-c104-4971-a960-86f234b26a90-tmp\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.417143 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:18.417063 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 17:44:18.417143 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:18.417084 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 17:44:18.417143 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:18.417096 2565 projected.go:196] Error preparing data for projected volume kube-api-access-dqwqs for pod openshift-network-diagnostics/network-check-target-slcq6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:44:18.417334 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:18.417188 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cea169f8-ef99-4ce1-9559-029a6507402f-kube-api-access-dqwqs podName:cea169f8-ef99-4ce1-9559-029a6507402f nodeName:}" failed. No retries permitted until 2026-07-23 17:44:18.917167913 +0000 UTC m=+2.995471246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dqwqs" (UniqueName: "kubernetes.io/projected/cea169f8-ef99-4ce1-9559-029a6507402f-kube-api-access-dqwqs") pod "network-check-target-slcq6" (UID: "cea169f8-ef99-4ce1-9559-029a6507402f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:44:18.419798 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.419775 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msflt\" (UniqueName: \"kubernetes.io/projected/84ce7a4f-1e92-434b-b939-616cce7d9188-kube-api-access-msflt\") pod \"ovnkube-node-pnj76\" (UID: \"84ce7a4f-1e92-434b-b939-616cce7d9188\") " pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.420996 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.420973 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sszl\" (UniqueName: \"kubernetes.io/projected/98b7a47e-3e44-401b-8d69-0052ef88824d-kube-api-access-6sszl\") pod \"aws-ebs-csi-driver-node-qk8n6\" (UID: \"98b7a47e-3e44-401b-8d69-0052ef88824d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk8n6" Jul 23 17:44:18.421396 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.421377 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjsc5\" (UniqueName: \"kubernetes.io/projected/bcc857f9-c104-4971-a960-86f234b26a90-kube-api-access-sjsc5\") pod \"tuned-5xjz4\" (UID: \"bcc857f9-c104-4971-a960-86f234b26a90\") " pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.511791 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.511753 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-system-cni-dir\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.511974 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.511801 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-etc-kubernetes\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.511974 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.511827 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/39d55095-246a-4ecf-bb04-36a9a8c92da4-kubelet-config\") pod \"global-pull-secret-syncer-2j6vd\" (UID: \"39d55095-246a-4ecf-bb04-36a9a8c92da4\") " pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:18.511974 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.511853 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b11ec86f-5925-4385-aeca-5b5e88f693fa-host\") pod \"node-ca-m8496\" (UID: \"b11ec86f-5925-4385-aeca-5b5e88f693fa\") " pod="openshift-image-registry/node-ca-m8496" Jul 23 17:44:18.511974 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.511880 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjnvl\" (UniqueName: \"kubernetes.io/projected/d7027def-1a58-4d31-a9ce-ea9814a550a3-kube-api-access-fjnvl\") pod \"network-metrics-daemon-t2hlf\" (UID: \"d7027def-1a58-4d31-a9ce-ea9814a550a3\") " pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:44:18.511974 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.511904 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rvs4\" (UniqueName: \"kubernetes.io/projected/b1112f49-7d09-4c67-b66d-29e03194d1f9-kube-api-access-8rvs4\") pod \"iptables-alerter-rtmx5\" (UID: \"b1112f49-7d09-4c67-b66d-29e03194d1f9\") " pod="openshift-network-operator/iptables-alerter-rtmx5" Jul 23 17:44:18.511974 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.511923 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-etc-kubernetes\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.511974 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.511930 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce-cni-binary-copy\") pod \"multus-additional-cni-plugins-vcfvv\" (UID: \"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvv" Jul 23 17:44:18.512324 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.511991 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7027def-1a58-4d31-a9ce-ea9814a550a3-metrics-certs\") pod \"network-metrics-daemon-t2hlf\" (UID: \"d7027def-1a58-4d31-a9ce-ea9814a550a3\") " pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:44:18.512324 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.512032 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-os-release\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.512324 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.512058 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-host-var-lib-cni-bin\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.512324 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.512083 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b1112f49-7d09-4c67-b66d-29e03194d1f9-iptables-alerter-script\") pod \"iptables-alerter-rtmx5\" (UID: \"b1112f49-7d09-4c67-b66d-29e03194d1f9\") " pod="openshift-network-operator/iptables-alerter-rtmx5" Jul 23 17:44:18.512324 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.512109 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-multus-socket-dir-parent\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.512324 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.512151 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2z7m\" (UniqueName: \"kubernetes.io/projected/69063b0f-0422-463f-833c-5aecf8d331a7-kube-api-access-k2z7m\") pod \"node-resolver-6nsmg\" (UID: \"69063b0f-0422-463f-833c-5aecf8d331a7\") " pod="openshift-dns/node-resolver-6nsmg" Jul 23 17:44:18.512324 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.512177 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-cnibin\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.512324 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.512188 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b11ec86f-5925-4385-aeca-5b5e88f693fa-host\") pod \"node-ca-m8496\" (UID: \"b11ec86f-5925-4385-aeca-5b5e88f693fa\") " pod="openshift-image-registry/node-ca-m8496" Jul 23 17:44:18.512324 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.512199 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6b9t\" (UniqueName: \"kubernetes.io/projected/91f8ae79-5ace-4707-82f0-c5a5770eba19-kube-api-access-m6b9t\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.512324 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.512222 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1112f49-7d09-4c67-b66d-29e03194d1f9-host-slash\") pod \"iptables-alerter-rtmx5\" (UID: \"b1112f49-7d09-4c67-b66d-29e03194d1f9\") " pod="openshift-network-operator/iptables-alerter-rtmx5" Jul 23 17:44:18.512324 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.512250 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-multus-cni-dir\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.512324 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.512255 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/39d55095-246a-4ecf-bb04-36a9a8c92da4-kubelet-config\") pod \"global-pull-secret-syncer-2j6vd\" (UID: \"39d55095-246a-4ecf-bb04-36a9a8c92da4\") " pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:18.512324 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.512277 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-host-var-lib-cni-multus\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.512324 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.512302 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce-cnibin\") pod \"multus-additional-cni-plugins-vcfvv\" (UID: \"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvv" Jul 23 17:44:18.512324 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.512325 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vcfvv\" (UID: \"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvv" Jul 23 17:44:18.513051 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.512349 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-host-var-lib-kubelet\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.513051 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.512371 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-multus-conf-dir\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.513051 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.512394 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39d55095-246a-4ecf-bb04-36a9a8c92da4-original-pull-secret\") pod \"global-pull-secret-syncer-2j6vd\" (UID: \"39d55095-246a-4ecf-bb04-36a9a8c92da4\") " pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:18.513051 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.512453 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-system-cni-dir\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.513051 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:18.512471 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 17:44:18.513051 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:18.512516 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:44:18.513051 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:18.512532 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d55095-246a-4ecf-bb04-36a9a8c92da4-original-pull-secret podName:39d55095-246a-4ecf-bb04-36a9a8c92da4 nodeName:}" failed. No retries permitted until 2026-07-23 17:44:19.012515112 +0000 UTC m=+3.090818448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39d55095-246a-4ecf-bb04-36a9a8c92da4-original-pull-secret") pod "global-pull-secret-syncer-2j6vd" (UID: "39d55095-246a-4ecf-bb04-36a9a8c92da4") : object "kube-system"/"original-pull-secret" not registered Jul 23 17:44:18.513051 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:18.512548 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7027def-1a58-4d31-a9ce-ea9814a550a3-metrics-certs podName:d7027def-1a58-4d31-a9ce-ea9814a550a3 nodeName:}" failed. No retries permitted until 2026-07-23 17:44:19.012538236 +0000 UTC m=+3.090841571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7027def-1a58-4d31-a9ce-ea9814a550a3-metrics-certs") pod "network-metrics-daemon-t2hlf" (UID: "d7027def-1a58-4d31-a9ce-ea9814a550a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:44:18.513051 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.512563 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce-cnibin\") pod \"multus-additional-cni-plugins-vcfvv\" (UID: \"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvv" Jul 23 17:44:18.513051 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.512621 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-os-release\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.513051 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.512831 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vcfvv\" (UID: \"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvv" Jul 23 17:44:18.513051 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.512851 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1112f49-7d09-4c67-b66d-29e03194d1f9-host-slash\") pod \"iptables-alerter-rtmx5\" (UID: \"b1112f49-7d09-4c67-b66d-29e03194d1f9\") " pod="openshift-network-operator/iptables-alerter-rtmx5" Jul 23 17:44:18.513051 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.512879 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-host-var-lib-kubelet\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.513051 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.512925 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-multus-conf-dir\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.513051 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.512960 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-multus-cni-dir\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.513051 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.512970 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-cnibin\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.513051 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.512988 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-host-var-lib-cni-multus\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.513861 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.513000 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b11ec86f-5925-4385-aeca-5b5e88f693fa-serviceca\") pod \"node-ca-m8496\" (UID: \"b11ec86f-5925-4385-aeca-5b5e88f693fa\") " pod="openshift-image-registry/node-ca-m8496" Jul 23 17:44:18.513861 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.513046 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-multus-socket-dir-parent\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.513861 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.513073 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcmjq\" (UniqueName: \"kubernetes.io/projected/b11ec86f-5925-4385-aeca-5b5e88f693fa-kube-api-access-rcmjq\") pod \"node-ca-m8496\" (UID: \"b11ec86f-5925-4385-aeca-5b5e88f693fa\") " pod="openshift-image-registry/node-ca-m8496" Jul 23 17:44:18.513861 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.513094 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce-cni-binary-copy\") pod \"multus-additional-cni-plugins-vcfvv\" (UID: \"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvv" Jul 23 17:44:18.513861 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.513101 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vcfvv\" (UID: \"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvv" Jul 23 17:44:18.513861 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.513126 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69063b0f-0422-463f-833c-5aecf8d331a7-tmp-dir\") pod \"node-resolver-6nsmg\" (UID: \"69063b0f-0422-463f-833c-5aecf8d331a7\") " pod="openshift-dns/node-resolver-6nsmg" Jul 23 17:44:18.513861 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.513127 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b1112f49-7d09-4c67-b66d-29e03194d1f9-iptables-alerter-script\") pod \"iptables-alerter-rtmx5\" (UID: \"b1112f49-7d09-4c67-b66d-29e03194d1f9\") " pod="openshift-network-operator/iptables-alerter-rtmx5" Jul 23 17:44:18.513861 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.513157 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/39d55095-246a-4ecf-bb04-36a9a8c92da4-dbus\") pod \"global-pull-secret-syncer-2j6vd\" (UID: \"39d55095-246a-4ecf-bb04-36a9a8c92da4\") " pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:18.513861 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.513180 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-hostroot\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.513861 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.513203 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/91f8ae79-5ace-4707-82f0-c5a5770eba19-multus-daemon-config\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.513861 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.513226 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-host-run-multus-certs\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.513861 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.513250 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce-os-release\") pod \"multus-additional-cni-plugins-vcfvv\" (UID: \"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvv" Jul 23 17:44:18.513861 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.513288 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-host-run-netns\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.513861 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.513314 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-hostroot\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.513861 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.513341 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b11ec86f-5925-4385-aeca-5b5e88f693fa-serviceca\") pod \"node-ca-m8496\" (UID: \"b11ec86f-5925-4385-aeca-5b5e88f693fa\") " pod="openshift-image-registry/node-ca-m8496" Jul 23 17:44:18.513861 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.513315 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbm6x\" (UniqueName: \"kubernetes.io/projected/5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce-kube-api-access-nbm6x\") pod \"multus-additional-cni-plugins-vcfvv\" (UID: \"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvv" Jul 23 17:44:18.513861 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.513426 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/69063b0f-0422-463f-833c-5aecf8d331a7-hosts-file\") pod \"node-resolver-6nsmg\" (UID: \"69063b0f-0422-463f-833c-5aecf8d331a7\") " pod="openshift-dns/node-resolver-6nsmg" Jul 23 17:44:18.513861 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.513438 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/39d55095-246a-4ecf-bb04-36a9a8c92da4-dbus\") pod \"global-pull-secret-syncer-2j6vd\" (UID: \"39d55095-246a-4ecf-bb04-36a9a8c92da4\") " pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:18.514701 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.513455 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91f8ae79-5ace-4707-82f0-c5a5770eba19-cni-binary-copy\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.514701 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.513477 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-host-run-multus-certs\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.514701 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.513482 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-host-run-k8s-cni-cncf-io\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.514701 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.513508 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce-system-cni-dir\") pod \"multus-additional-cni-plugins-vcfvv\" (UID: \"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvv" Jul 23 17:44:18.514701 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.513531 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce-os-release\") pod \"multus-additional-cni-plugins-vcfvv\" (UID: \"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvv" Jul 23 17:44:18.514701 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.513538 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vcfvv\" (UID: \"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvv" Jul 23 17:44:18.514701 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.513678 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-host-run-netns\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.514701 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.513804 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/91f8ae79-5ace-4707-82f0-c5a5770eba19-multus-daemon-config\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.514701 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.513886 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-host-var-lib-cni-bin\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.514701 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.513924 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69063b0f-0422-463f-833c-5aecf8d331a7-tmp-dir\") pod \"node-resolver-6nsmg\" (UID: \"69063b0f-0422-463f-833c-5aecf8d331a7\") " pod="openshift-dns/node-resolver-6nsmg" Jul 23 17:44:18.514701 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.513991 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vcfvv\" (UID: \"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvv" Jul 23 17:44:18.514701 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.513990 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/69063b0f-0422-463f-833c-5aecf8d331a7-hosts-file\") pod \"node-resolver-6nsmg\" (UID: \"69063b0f-0422-463f-833c-5aecf8d331a7\") " pod="openshift-dns/node-resolver-6nsmg" Jul 23 17:44:18.514701 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.514027 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/91f8ae79-5ace-4707-82f0-c5a5770eba19-host-run-k8s-cni-cncf-io\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.514701 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.514057 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce-system-cni-dir\") pod \"multus-additional-cni-plugins-vcfvv\" (UID: \"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvv" Jul 23 17:44:18.514701 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.514416 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91f8ae79-5ace-4707-82f0-c5a5770eba19-cni-binary-copy\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.514701 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.514507 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vcfvv\" (UID: \"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvv" Jul 23 17:44:18.525221 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.525081 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rvs4\" (UniqueName: \"kubernetes.io/projected/b1112f49-7d09-4c67-b66d-29e03194d1f9-kube-api-access-8rvs4\") pod \"iptables-alerter-rtmx5\" (UID: \"b1112f49-7d09-4c67-b66d-29e03194d1f9\") " pod="openshift-network-operator/iptables-alerter-rtmx5" Jul 23 17:44:18.525221 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.525089 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2z7m\" (UniqueName: \"kubernetes.io/projected/69063b0f-0422-463f-833c-5aecf8d331a7-kube-api-access-k2z7m\") pod \"node-resolver-6nsmg\" (UID: \"69063b0f-0422-463f-833c-5aecf8d331a7\") " pod="openshift-dns/node-resolver-6nsmg" Jul 23 17:44:18.525221 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.525089 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6b9t\" (UniqueName: \"kubernetes.io/projected/91f8ae79-5ace-4707-82f0-c5a5770eba19-kube-api-access-m6b9t\") pod \"multus-ggqsr\" (UID: \"91f8ae79-5ace-4707-82f0-c5a5770eba19\") " pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.525221 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.525145 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjnvl\" (UniqueName: \"kubernetes.io/projected/d7027def-1a58-4d31-a9ce-ea9814a550a3-kube-api-access-fjnvl\") pod \"network-metrics-daemon-t2hlf\" (UID: \"d7027def-1a58-4d31-a9ce-ea9814a550a3\") " pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:44:18.526985 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.526826 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbm6x\" (UniqueName: \"kubernetes.io/projected/5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce-kube-api-access-nbm6x\") pod \"multus-additional-cni-plugins-vcfvv\" (UID: \"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvv" Jul 23 17:44:18.527150 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.527124 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcmjq\" (UniqueName: \"kubernetes.io/projected/b11ec86f-5925-4385-aeca-5b5e88f693fa-kube-api-access-rcmjq\") pod \"node-ca-m8496\" (UID: \"b11ec86f-5925-4385-aeca-5b5e88f693fa\") " pod="openshift-image-registry/node-ca-m8496" Jul 23 17:44:18.603142 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.603109 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk8n6" Jul 23 17:44:18.612709 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:18.612659 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98b7a47e_3e44_401b_8d69_0052ef88824d.slice/crio-c7ff4abc833ff29373f647c7afa51213e5221a109c86259f17dbb7eab2ac0f2a WatchSource:0}: Error finding container c7ff4abc833ff29373f647c7afa51213e5221a109c86259f17dbb7eab2ac0f2a: Status 404 returned error can't find the container with id c7ff4abc833ff29373f647c7afa51213e5221a109c86259f17dbb7eab2ac0f2a Jul 23 17:44:18.613914 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.613883 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" Jul 23 17:44:18.623425 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:18.623350 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc857f9_c104_4971_a960_86f234b26a90.slice/crio-ab48d09d3ad2bedc3bad6af93942082577b63ea60b6b6856f288755589139a29 WatchSource:0}: Error finding container ab48d09d3ad2bedc3bad6af93942082577b63ea60b6b6856f288755589139a29: Status 404 returned error can't find the container with id ab48d09d3ad2bedc3bad6af93942082577b63ea60b6b6856f288755589139a29 Jul 23 17:44:18.624207 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.624185 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:18.631773 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.631751 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kcd4h" Jul 23 17:44:18.633259 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:18.633232 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84ce7a4f_1e92_434b_b939_616cce7d9188.slice/crio-13a92e0859b67840985d2082f3c9ee7ca4c4d78c6b6a91c1a1cce55d8bb4cd69 WatchSource:0}: Error finding container 13a92e0859b67840985d2082f3c9ee7ca4c4d78c6b6a91c1a1cce55d8bb4cd69: Status 404 returned error can't find the container with id 13a92e0859b67840985d2082f3c9ee7ca4c4d78c6b6a91c1a1cce55d8bb4cd69 Jul 23 17:44:18.639199 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:18.639173 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a324aac_61f5_46a9_b847_15730ba25434.slice/crio-8a6a9c9163c6d2281712aab0ea3b1f3d6ae6cd34c998236ae805bc60914171a5 WatchSource:0}: Error finding container 8a6a9c9163c6d2281712aab0ea3b1f3d6ae6cd34c998236ae805bc60914171a5: Status 404 returned error can't find the container with id 8a6a9c9163c6d2281712aab0ea3b1f3d6ae6cd34c998236ae805bc60914171a5 Jul 23 17:44:18.639855 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.639834 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6nsmg" Jul 23 17:44:18.648002 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:18.647974 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69063b0f_0422_463f_833c_5aecf8d331a7.slice/crio-f07136cc9e64553baebd89c8cea8aa1aa1743dda903afc7ea62987db853fd74f WatchSource:0}: Error finding container f07136cc9e64553baebd89c8cea8aa1aa1743dda903afc7ea62987db853fd74f: Status 404 returned error can't find the container with id f07136cc9e64553baebd89c8cea8aa1aa1743dda903afc7ea62987db853fd74f Jul 23 17:44:18.648798 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.648777 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m8496" Jul 23 17:44:18.657811 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:18.657779 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb11ec86f_5925_4385_aeca_5b5e88f693fa.slice/crio-f2e4976fe86e2d42be9f49f24933d3e444488fb32fa84a679657b31f23413251 WatchSource:0}: Error finding container f2e4976fe86e2d42be9f49f24933d3e444488fb32fa84a679657b31f23413251: Status 404 returned error can't find the container with id f2e4976fe86e2d42be9f49f24933d3e444488fb32fa84a679657b31f23413251 Jul 23 17:44:18.658645 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.658470 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vcfvv" Jul 23 17:44:18.664474 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.664453 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ggqsr" Jul 23 17:44:18.672980 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.672955 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rtmx5" Jul 23 17:44:18.676436 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:18.676404 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f8ae79_5ace_4707_82f0_c5a5770eba19.slice/crio-62293228aeb370d4fe665ca0e84bd651e3ec7dec6abff1ea31165481ee4b29f2 WatchSource:0}: Error finding container 62293228aeb370d4fe665ca0e84bd651e3ec7dec6abff1ea31165481ee4b29f2: Status 404 returned error can't find the container with id 62293228aeb370d4fe665ca0e84bd651e3ec7dec6abff1ea31165481ee4b29f2 Jul 23 17:44:18.681422 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:18.681398 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1112f49_7d09_4c67_b66d_29e03194d1f9.slice/crio-332efe5415921e656583cf462bcbdcfcc25bc0ee31cc3917cfc588d020fcaa09 WatchSource:0}: Error finding container 332efe5415921e656583cf462bcbdcfcc25bc0ee31cc3917cfc588d020fcaa09: Status 404 returned error can't find the container with id 332efe5415921e656583cf462bcbdcfcc25bc0ee31cc3917cfc588d020fcaa09 Jul 23 17:44:18.829588 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:18.829505 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 17:44:19.017526 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:19.017488 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7027def-1a58-4d31-a9ce-ea9814a550a3-metrics-certs\") pod \"network-metrics-daemon-t2hlf\" (UID: \"d7027def-1a58-4d31-a9ce-ea9814a550a3\") " pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:44:19.017526 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:19.017530 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqwqs\" (UniqueName: \"kubernetes.io/projected/cea169f8-ef99-4ce1-9559-029a6507402f-kube-api-access-dqwqs\") pod \"network-check-target-slcq6\" (UID: \"cea169f8-ef99-4ce1-9559-029a6507402f\") " pod="openshift-network-diagnostics/network-check-target-slcq6" Jul 23 17:44:19.017736 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:19.017557 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39d55095-246a-4ecf-bb04-36a9a8c92da4-original-pull-secret\") pod \"global-pull-secret-syncer-2j6vd\" (UID: \"39d55095-246a-4ecf-bb04-36a9a8c92da4\") " pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:19.017736 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:19.017658 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:44:19.017736 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:19.017672 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 17:44:19.017736 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:19.017678 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 17:44:19.017736 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:19.017692 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 17:44:19.017736 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:19.017701 2565 projected.go:196] Error preparing data for projected volume kube-api-access-dqwqs for pod openshift-network-diagnostics/network-check-target-slcq6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:44:19.017736 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:19.017731 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d55095-246a-4ecf-bb04-36a9a8c92da4-original-pull-secret podName:39d55095-246a-4ecf-bb04-36a9a8c92da4 nodeName:}" failed. No retries permitted until 2026-07-23 17:44:20.017716311 +0000 UTC m=+4.096019642 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39d55095-246a-4ecf-bb04-36a9a8c92da4-original-pull-secret") pod "global-pull-secret-syncer-2j6vd" (UID: "39d55095-246a-4ecf-bb04-36a9a8c92da4") : object "kube-system"/"original-pull-secret" not registered Jul 23 17:44:19.017973 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:19.017749 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7027def-1a58-4d31-a9ce-ea9814a550a3-metrics-certs podName:d7027def-1a58-4d31-a9ce-ea9814a550a3 nodeName:}" failed. No retries permitted until 2026-07-23 17:44:20.017741315 +0000 UTC m=+4.096044646 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7027def-1a58-4d31-a9ce-ea9814a550a3-metrics-certs") pod "network-metrics-daemon-t2hlf" (UID: "d7027def-1a58-4d31-a9ce-ea9814a550a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:44:19.017973 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:19.017762 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cea169f8-ef99-4ce1-9559-029a6507402f-kube-api-access-dqwqs podName:cea169f8-ef99-4ce1-9559-029a6507402f nodeName:}" failed. No retries permitted until 2026-07-23 17:44:20.017754222 +0000 UTC m=+4.096057553 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dqwqs" (UniqueName: "kubernetes.io/projected/cea169f8-ef99-4ce1-9559-029a6507402f-kube-api-access-dqwqs") pod "network-check-target-slcq6" (UID: "cea169f8-ef99-4ce1-9559-029a6507402f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:44:19.339309 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:19.339268 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 17:39:18 +0000 UTC" deadline="2028-01-17 00:02:10.999489053 +0000 UTC" Jul 23 17:44:19.339309 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:19.339310 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13014h17m51.660187651s" Jul 23 17:44:19.408522 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:19.408483 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ggqsr" event={"ID":"91f8ae79-5ace-4707-82f0-c5a5770eba19","Type":"ContainerStarted","Data":"62293228aeb370d4fe665ca0e84bd651e3ec7dec6abff1ea31165481ee4b29f2"} Jul 23 17:44:19.409720 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:19.409696 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m8496" event={"ID":"b11ec86f-5925-4385-aeca-5b5e88f693fa","Type":"ContainerStarted","Data":"f2e4976fe86e2d42be9f49f24933d3e444488fb32fa84a679657b31f23413251"} Jul 23 17:44:19.410919 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:19.410896 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kcd4h" event={"ID":"4a324aac-61f5-46a9-b847-15730ba25434","Type":"ContainerStarted","Data":"8a6a9c9163c6d2281712aab0ea3b1f3d6ae6cd34c998236ae805bc60914171a5"} Jul 23 17:44:19.412159 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:19.412134 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk8n6" event={"ID":"98b7a47e-3e44-401b-8d69-0052ef88824d","Type":"ContainerStarted","Data":"c7ff4abc833ff29373f647c7afa51213e5221a109c86259f17dbb7eab2ac0f2a"} Jul 23 17:44:19.413451 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:19.413425 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rtmx5" event={"ID":"b1112f49-7d09-4c67-b66d-29e03194d1f9","Type":"ContainerStarted","Data":"332efe5415921e656583cf462bcbdcfcc25bc0ee31cc3917cfc588d020fcaa09"} Jul 23 17:44:19.414509 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:19.414485 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcfvv" event={"ID":"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce","Type":"ContainerStarted","Data":"6ac7627c7cc99232d382649fc2e41bec08f9d55c27d1a04217082cc549fe19d1"} Jul 23 17:44:19.415573 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:19.415540 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6nsmg" event={"ID":"69063b0f-0422-463f-833c-5aecf8d331a7","Type":"ContainerStarted","Data":"f07136cc9e64553baebd89c8cea8aa1aa1743dda903afc7ea62987db853fd74f"} Jul 23 17:44:19.416718 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:19.416693 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" event={"ID":"84ce7a4f-1e92-434b-b939-616cce7d9188","Type":"ContainerStarted","Data":"13a92e0859b67840985d2082f3c9ee7ca4c4d78c6b6a91c1a1cce55d8bb4cd69"} Jul 23 17:44:19.417872 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:19.417851 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" event={"ID":"bcc857f9-c104-4971-a960-86f234b26a90","Type":"ContainerStarted","Data":"ab48d09d3ad2bedc3bad6af93942082577b63ea60b6b6856f288755589139a29"} Jul 23 17:44:20.027032 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:20.026723 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqwqs\" (UniqueName: \"kubernetes.io/projected/cea169f8-ef99-4ce1-9559-029a6507402f-kube-api-access-dqwqs\") pod \"network-check-target-slcq6\" (UID: \"cea169f8-ef99-4ce1-9559-029a6507402f\") " pod="openshift-network-diagnostics/network-check-target-slcq6" Jul 23 17:44:20.027218 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:20.027104 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39d55095-246a-4ecf-bb04-36a9a8c92da4-original-pull-secret\") pod \"global-pull-secret-syncer-2j6vd\" (UID: \"39d55095-246a-4ecf-bb04-36a9a8c92da4\") " pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:20.027218 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:20.027179 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7027def-1a58-4d31-a9ce-ea9814a550a3-metrics-certs\") pod \"network-metrics-daemon-t2hlf\" (UID: \"d7027def-1a58-4d31-a9ce-ea9814a550a3\") " pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:44:20.027340 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:20.026962 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 17:44:20.027340 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:20.027322 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 17:44:20.027436 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:20.027346 2565 projected.go:196] Error preparing data for projected volume kube-api-access-dqwqs for pod openshift-network-diagnostics/network-check-target-slcq6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:44:20.027436 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:20.027424 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cea169f8-ef99-4ce1-9559-029a6507402f-kube-api-access-dqwqs podName:cea169f8-ef99-4ce1-9559-029a6507402f nodeName:}" failed. No retries permitted until 2026-07-23 17:44:22.027390806 +0000 UTC m=+6.105694155 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dqwqs" (UniqueName: "kubernetes.io/projected/cea169f8-ef99-4ce1-9559-029a6507402f-kube-api-access-dqwqs") pod "network-check-target-slcq6" (UID: "cea169f8-ef99-4ce1-9559-029a6507402f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:44:20.027885 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:20.027866 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 17:44:20.027992 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:20.027934 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d55095-246a-4ecf-bb04-36a9a8c92da4-original-pull-secret podName:39d55095-246a-4ecf-bb04-36a9a8c92da4 nodeName:}" failed. No retries permitted until 2026-07-23 17:44:22.027909767 +0000 UTC m=+6.106213099 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39d55095-246a-4ecf-bb04-36a9a8c92da4-original-pull-secret") pod "global-pull-secret-syncer-2j6vd" (UID: "39d55095-246a-4ecf-bb04-36a9a8c92da4") : object "kube-system"/"original-pull-secret" not registered Jul 23 17:44:20.027992 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:20.027294 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:44:20.027992 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:20.027973 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7027def-1a58-4d31-a9ce-ea9814a550a3-metrics-certs podName:d7027def-1a58-4d31-a9ce-ea9814a550a3 nodeName:}" failed. No retries permitted until 2026-07-23 17:44:22.02796348 +0000 UTC m=+6.106266824 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7027def-1a58-4d31-a9ce-ea9814a550a3-metrics-certs") pod "network-metrics-daemon-t2hlf" (UID: "d7027def-1a58-4d31-a9ce-ea9814a550a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:44:20.339841 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:20.339754 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 17:39:18 +0000 UTC" deadline="2028-04-08 12:04:11.278692968 +0000 UTC" Jul 23 17:44:20.339841 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:20.339793 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14994h19m50.938904615s" Jul 23 17:44:20.410758 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:20.410727 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:20.410935 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:20.410894 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2j6vd" podUID="39d55095-246a-4ecf-bb04-36a9a8c92da4" Jul 23 17:44:20.411005 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:20.410989 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slcq6" Jul 23 17:44:20.411109 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:20.411093 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slcq6" podUID="cea169f8-ef99-4ce1-9559-029a6507402f" Jul 23 17:44:20.411584 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:20.411565 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:44:20.411790 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:20.411769 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2hlf" podUID="d7027def-1a58-4d31-a9ce-ea9814a550a3" Jul 23 17:44:20.430418 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:20.430384 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-82.ec2.internal" event={"ID":"175ceae0d598475e91b574d725c1c32d","Type":"ContainerStarted","Data":"96a5c5fe31190c1d342a8f396aba816f04772948df7d1be7006afd91714137f6"} Jul 23 17:44:20.436302 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:20.436262 2565 generic.go:356] "Generic (PLEG): container finished" podID="4caf25fc2a9f5a4108b00096ab025031" containerID="d1c6d55c3bc46be12bf71ce8a865f16bcc1b531cc5916ed3973ce7aef8949cec" exitCode=0 Jul 23 17:44:20.436461 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:20.436309 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-82.ec2.internal" event={"ID":"4caf25fc2a9f5a4108b00096ab025031","Type":"ContainerDied","Data":"d1c6d55c3bc46be12bf71ce8a865f16bcc1b531cc5916ed3973ce7aef8949cec"} Jul 23 17:44:20.464183 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:20.464132 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-82.ec2.internal" podStartSLOduration=3.464112593 podStartE2EDuration="3.464112593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:44:20.447770552 +0000 UTC m=+4.526073921" watchObservedRunningTime="2026-07-23 17:44:20.464112593 +0000 UTC m=+4.542415946" Jul 23 17:44:21.462346 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:21.462244 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-82.ec2.internal" event={"ID":"4caf25fc2a9f5a4108b00096ab025031","Type":"ContainerStarted","Data":"8e5baa1ddf3c532a5aa8dbd6db0540a6a9ab84d770b48b4b294d545a42a351f7"} Jul 23 17:44:22.044609 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:22.043757 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7027def-1a58-4d31-a9ce-ea9814a550a3-metrics-certs\") pod \"network-metrics-daemon-t2hlf\" (UID: \"d7027def-1a58-4d31-a9ce-ea9814a550a3\") " pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:44:22.044609 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:22.043813 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqwqs\" (UniqueName: \"kubernetes.io/projected/cea169f8-ef99-4ce1-9559-029a6507402f-kube-api-access-dqwqs\") pod \"network-check-target-slcq6\" (UID: \"cea169f8-ef99-4ce1-9559-029a6507402f\") " pod="openshift-network-diagnostics/network-check-target-slcq6" Jul 23 17:44:22.044609 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:22.043850 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39d55095-246a-4ecf-bb04-36a9a8c92da4-original-pull-secret\") pod \"global-pull-secret-syncer-2j6vd\" (UID: \"39d55095-246a-4ecf-bb04-36a9a8c92da4\") " pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:22.044609 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:22.043985 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 17:44:22.044609 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:22.044046 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d55095-246a-4ecf-bb04-36a9a8c92da4-original-pull-secret podName:39d55095-246a-4ecf-bb04-36a9a8c92da4 nodeName:}" failed. No retries permitted until 2026-07-23 17:44:26.044026813 +0000 UTC m=+10.122330149 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39d55095-246a-4ecf-bb04-36a9a8c92da4-original-pull-secret") pod "global-pull-secret-syncer-2j6vd" (UID: "39d55095-246a-4ecf-bb04-36a9a8c92da4") : object "kube-system"/"original-pull-secret" not registered Jul 23 17:44:22.044609 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:22.044106 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:44:22.044609 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:22.044136 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7027def-1a58-4d31-a9ce-ea9814a550a3-metrics-certs podName:d7027def-1a58-4d31-a9ce-ea9814a550a3 nodeName:}" failed. No retries permitted until 2026-07-23 17:44:26.044125834 +0000 UTC m=+10.122429167 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7027def-1a58-4d31-a9ce-ea9814a550a3-metrics-certs") pod "network-metrics-daemon-t2hlf" (UID: "d7027def-1a58-4d31-a9ce-ea9814a550a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:44:22.044609 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:22.044213 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 17:44:22.044609 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:22.044227 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 17:44:22.044609 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:22.044240 2565 projected.go:196] Error preparing data for projected volume kube-api-access-dqwqs for pod openshift-network-diagnostics/network-check-target-slcq6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:44:22.044609 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:22.044273 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cea169f8-ef99-4ce1-9559-029a6507402f-kube-api-access-dqwqs podName:cea169f8-ef99-4ce1-9559-029a6507402f nodeName:}" failed. No retries permitted until 2026-07-23 17:44:26.044262912 +0000 UTC m=+10.122566246 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dqwqs" (UniqueName: "kubernetes.io/projected/cea169f8-ef99-4ce1-9559-029a6507402f-kube-api-access-dqwqs") pod "network-check-target-slcq6" (UID: "cea169f8-ef99-4ce1-9559-029a6507402f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:44:22.402765 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:22.402688 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:44:22.402929 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:22.402831 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2hlf" podUID="d7027def-1a58-4d31-a9ce-ea9814a550a3" Jul 23 17:44:22.403218 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:22.403193 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slcq6" Jul 23 17:44:22.403310 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:22.403297 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slcq6" podUID="cea169f8-ef99-4ce1-9559-029a6507402f" Jul 23 17:44:22.403379 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:22.403345 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:22.403435 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:22.403412 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2j6vd" podUID="39d55095-246a-4ecf-bb04-36a9a8c92da4" Jul 23 17:44:24.400526 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:24.400494 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slcq6" Jul 23 17:44:24.400526 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:24.400522 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:24.401065 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:24.400640 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slcq6" podUID="cea169f8-ef99-4ce1-9559-029a6507402f" Jul 23 17:44:24.401065 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:24.400490 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:44:24.401065 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:24.400836 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2hlf" podUID="d7027def-1a58-4d31-a9ce-ea9814a550a3" Jul 23 17:44:24.401065 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:24.400947 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2j6vd" podUID="39d55095-246a-4ecf-bb04-36a9a8c92da4" Jul 23 17:44:26.081972 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:26.081020 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39d55095-246a-4ecf-bb04-36a9a8c92da4-original-pull-secret\") pod \"global-pull-secret-syncer-2j6vd\" (UID: \"39d55095-246a-4ecf-bb04-36a9a8c92da4\") " pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:26.081972 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:26.081117 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7027def-1a58-4d31-a9ce-ea9814a550a3-metrics-certs\") pod \"network-metrics-daemon-t2hlf\" (UID: \"d7027def-1a58-4d31-a9ce-ea9814a550a3\") " pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:44:26.081972 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:26.081156 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqwqs\" (UniqueName: \"kubernetes.io/projected/cea169f8-ef99-4ce1-9559-029a6507402f-kube-api-access-dqwqs\") pod \"network-check-target-slcq6\" (UID: \"cea169f8-ef99-4ce1-9559-029a6507402f\") " pod="openshift-network-diagnostics/network-check-target-slcq6" Jul 23 17:44:26.081972 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:26.081301 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 17:44:26.081972 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:26.081325 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 17:44:26.081972 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:26.081337 2565 projected.go:196] Error preparing data for projected volume kube-api-access-dqwqs for pod openshift-network-diagnostics/network-check-target-slcq6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:44:26.081972 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:26.081395 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cea169f8-ef99-4ce1-9559-029a6507402f-kube-api-access-dqwqs podName:cea169f8-ef99-4ce1-9559-029a6507402f nodeName:}" failed. No retries permitted until 2026-07-23 17:44:34.081375933 +0000 UTC m=+18.159679282 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dqwqs" (UniqueName: "kubernetes.io/projected/cea169f8-ef99-4ce1-9559-029a6507402f-kube-api-access-dqwqs") pod "network-check-target-slcq6" (UID: "cea169f8-ef99-4ce1-9559-029a6507402f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:44:26.081972 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:26.081467 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 17:44:26.081972 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:26.081503 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d55095-246a-4ecf-bb04-36a9a8c92da4-original-pull-secret podName:39d55095-246a-4ecf-bb04-36a9a8c92da4 nodeName:}" failed. No retries permitted until 2026-07-23 17:44:34.081491606 +0000 UTC m=+18.159794942 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39d55095-246a-4ecf-bb04-36a9a8c92da4-original-pull-secret") pod "global-pull-secret-syncer-2j6vd" (UID: "39d55095-246a-4ecf-bb04-36a9a8c92da4") : object "kube-system"/"original-pull-secret" not registered Jul 23 17:44:26.081972 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:26.081560 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:44:26.081972 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:26.081610 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7027def-1a58-4d31-a9ce-ea9814a550a3-metrics-certs podName:d7027def-1a58-4d31-a9ce-ea9814a550a3 nodeName:}" failed. No retries permitted until 2026-07-23 17:44:34.081581789 +0000 UTC m=+18.159885129 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7027def-1a58-4d31-a9ce-ea9814a550a3-metrics-certs") pod "network-metrics-daemon-t2hlf" (UID: "d7027def-1a58-4d31-a9ce-ea9814a550a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:44:26.401067 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:26.400988 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slcq6" Jul 23 17:44:26.401220 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:26.401092 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slcq6" podUID="cea169f8-ef99-4ce1-9559-029a6507402f" Jul 23 17:44:26.401220 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:26.401182 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:26.401343 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:26.401265 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2j6vd" podUID="39d55095-246a-4ecf-bb04-36a9a8c92da4" Jul 23 17:44:26.401343 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:26.401318 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:44:26.401479 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:26.401388 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2hlf" podUID="d7027def-1a58-4d31-a9ce-ea9814a550a3" Jul 23 17:44:28.399557 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:28.399528 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slcq6" Jul 23 17:44:28.400041 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:28.399665 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slcq6" podUID="cea169f8-ef99-4ce1-9559-029a6507402f" Jul 23 17:44:28.400041 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:28.399677 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:28.400041 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:28.399783 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2j6vd" podUID="39d55095-246a-4ecf-bb04-36a9a8c92da4" Jul 23 17:44:28.400229 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:28.400210 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:44:28.400334 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:28.400312 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2hlf" podUID="d7027def-1a58-4d31-a9ce-ea9814a550a3" Jul 23 17:44:30.399968 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:30.399929 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:30.400380 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:30.399929 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slcq6" Jul 23 17:44:30.400380 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:30.400065 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2j6vd" podUID="39d55095-246a-4ecf-bb04-36a9a8c92da4" Jul 23 17:44:30.400380 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:30.400120 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slcq6" podUID="cea169f8-ef99-4ce1-9559-029a6507402f" Jul 23 17:44:30.400380 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:30.399937 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:44:30.400380 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:30.400251 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2hlf" podUID="d7027def-1a58-4d31-a9ce-ea9814a550a3" Jul 23 17:44:32.400313 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:32.400257 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:32.400313 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:32.400288 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slcq6" Jul 23 17:44:32.400900 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:32.400260 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:44:32.400900 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:32.400391 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2j6vd" podUID="39d55095-246a-4ecf-bb04-36a9a8c92da4" Jul 23 17:44:32.400900 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:32.400460 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2hlf" podUID="d7027def-1a58-4d31-a9ce-ea9814a550a3" Jul 23 17:44:32.400900 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:32.400545 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slcq6" podUID="cea169f8-ef99-4ce1-9559-029a6507402f" Jul 23 17:44:34.144430 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:34.144389 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7027def-1a58-4d31-a9ce-ea9814a550a3-metrics-certs\") pod \"network-metrics-daemon-t2hlf\" (UID: \"d7027def-1a58-4d31-a9ce-ea9814a550a3\") " pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:44:34.144959 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:34.144448 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqwqs\" (UniqueName: \"kubernetes.io/projected/cea169f8-ef99-4ce1-9559-029a6507402f-kube-api-access-dqwqs\") pod \"network-check-target-slcq6\" (UID: \"cea169f8-ef99-4ce1-9559-029a6507402f\") " pod="openshift-network-diagnostics/network-check-target-slcq6" Jul 23 17:44:34.144959 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:34.144493 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39d55095-246a-4ecf-bb04-36a9a8c92da4-original-pull-secret\") pod \"global-pull-secret-syncer-2j6vd\" (UID: \"39d55095-246a-4ecf-bb04-36a9a8c92da4\") " pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:34.144959 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:34.144613 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 17:44:34.144959 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:34.144613 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:44:34.144959 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:34.144692 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d55095-246a-4ecf-bb04-36a9a8c92da4-original-pull-secret podName:39d55095-246a-4ecf-bb04-36a9a8c92da4 nodeName:}" failed. No retries permitted until 2026-07-23 17:44:50.144672476 +0000 UTC m=+34.222975838 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39d55095-246a-4ecf-bb04-36a9a8c92da4-original-pull-secret") pod "global-pull-secret-syncer-2j6vd" (UID: "39d55095-246a-4ecf-bb04-36a9a8c92da4") : object "kube-system"/"original-pull-secret" not registered Jul 23 17:44:34.144959 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:34.144734 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7027def-1a58-4d31-a9ce-ea9814a550a3-metrics-certs podName:d7027def-1a58-4d31-a9ce-ea9814a550a3 nodeName:}" failed. No retries permitted until 2026-07-23 17:44:50.144715234 +0000 UTC m=+34.223018566 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7027def-1a58-4d31-a9ce-ea9814a550a3-metrics-certs") pod "network-metrics-daemon-t2hlf" (UID: "d7027def-1a58-4d31-a9ce-ea9814a550a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:44:34.144959 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:34.144668 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 17:44:34.144959 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:34.144766 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 17:44:34.144959 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:34.144783 2565 projected.go:196] Error preparing data for projected volume kube-api-access-dqwqs for pod openshift-network-diagnostics/network-check-target-slcq6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:44:34.144959 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:34.144820 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cea169f8-ef99-4ce1-9559-029a6507402f-kube-api-access-dqwqs podName:cea169f8-ef99-4ce1-9559-029a6507402f nodeName:}" failed. No retries permitted until 2026-07-23 17:44:50.144809387 +0000 UTC m=+34.223112719 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dqwqs" (UniqueName: "kubernetes.io/projected/cea169f8-ef99-4ce1-9559-029a6507402f-kube-api-access-dqwqs") pod "network-check-target-slcq6" (UID: "cea169f8-ef99-4ce1-9559-029a6507402f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:44:34.400322 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:34.400207 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:34.400322 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:34.400277 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:44:34.400548 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:34.400376 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2hlf" podUID="d7027def-1a58-4d31-a9ce-ea9814a550a3" Jul 23 17:44:34.400548 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:34.400378 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slcq6" Jul 23 17:44:34.400548 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:34.400502 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2j6vd" podUID="39d55095-246a-4ecf-bb04-36a9a8c92da4" Jul 23 17:44:34.400727 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:34.400587 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slcq6" podUID="cea169f8-ef99-4ce1-9559-029a6507402f" Jul 23 17:44:36.400664 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:36.400621 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:44:36.400664 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:36.400665 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slcq6" Jul 23 17:44:36.401164 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:36.400740 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:36.401164 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:36.400771 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2hlf" podUID="d7027def-1a58-4d31-a9ce-ea9814a550a3" Jul 23 17:44:36.401164 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:36.400818 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2j6vd" podUID="39d55095-246a-4ecf-bb04-36a9a8c92da4" Jul 23 17:44:36.401164 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:36.400942 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slcq6" podUID="cea169f8-ef99-4ce1-9559-029a6507402f" Jul 23 17:44:38.400638 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:38.400333 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:44:38.401430 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:38.400340 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slcq6" Jul 23 17:44:38.401430 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:38.400748 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2hlf" podUID="d7027def-1a58-4d31-a9ce-ea9814a550a3" Jul 23 17:44:38.401430 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:38.400813 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slcq6" podUID="cea169f8-ef99-4ce1-9559-029a6507402f" Jul 23 17:44:38.401430 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:38.400368 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:38.401430 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:38.400928 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2j6vd" podUID="39d55095-246a-4ecf-bb04-36a9a8c92da4" Jul 23 17:44:38.497851 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:38.497817 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ggqsr" event={"ID":"91f8ae79-5ace-4707-82f0-c5a5770eba19","Type":"ContainerStarted","Data":"105324ff3264ee242a7bc02c63aaa01c49c1568d5f10e6f54fa26e3fccaa688e"} Jul 23 17:44:38.499226 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:38.499203 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m8496" event={"ID":"b11ec86f-5925-4385-aeca-5b5e88f693fa","Type":"ContainerStarted","Data":"b364cf12b43b7c946866e3c998170dd49d8c1073c9c8c544dc79c9d9c085c2bd"} Jul 23 17:44:38.500610 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:38.500567 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kcd4h" event={"ID":"4a324aac-61f5-46a9-b847-15730ba25434","Type":"ContainerStarted","Data":"66fbd523ab2a3df6a93b594e626eb22f50a27ca4664cba9e5e8f918d76d015c9"} Jul 23 17:44:38.501933 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:38.501916 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk8n6" event={"ID":"98b7a47e-3e44-401b-8d69-0052ef88824d","Type":"ContainerStarted","Data":"8bd03196958deedb6f9ac6f01fb8547d057a957e1bcebb90e95af042bc28e34a"} Jul 23 17:44:38.503485 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:38.503453 2565 generic.go:356] "Generic (PLEG): container finished" podID="5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce" containerID="f591fee8ed0e23af1cd91075ccb48ee7db9904031babef17c010442b66ef8f7d" exitCode=0 Jul 23 17:44:38.503575 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:38.503536 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcfvv" event={"ID":"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce","Type":"ContainerDied","Data":"f591fee8ed0e23af1cd91075ccb48ee7db9904031babef17c010442b66ef8f7d"} Jul 23 17:44:38.505156 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:38.505103 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6nsmg" event={"ID":"69063b0f-0422-463f-833c-5aecf8d331a7","Type":"ContainerStarted","Data":"e1bf9daeb1c3b76cb9db1525a706451da71b541ad5e69caac9441560af1b9233"} Jul 23 17:44:38.507268 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:38.507243 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" event={"ID":"84ce7a4f-1e92-434b-b939-616cce7d9188","Type":"ContainerStarted","Data":"b8043d7f70ab6272a77750546c788ec532ada0dfc4d2132f2fa73875296c6848"} Jul 23 17:44:38.507359 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:38.507317 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" event={"ID":"84ce7a4f-1e92-434b-b939-616cce7d9188","Type":"ContainerStarted","Data":"b70e7b120c1c021031676b8ba5391946d986abaf7a9a767cb3ec7bf39954a6e0"} Jul 23 17:44:38.507359 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:38.507350 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" event={"ID":"84ce7a4f-1e92-434b-b939-616cce7d9188","Type":"ContainerStarted","Data":"d1d2310eae67e162df64fc0ddbaa87ec442ca292690f08cd6a58720623e6068b"} Jul 23 17:44:38.508388 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:38.508372 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" event={"ID":"bcc857f9-c104-4971-a960-86f234b26a90","Type":"ContainerStarted","Data":"44a815c7551cc9be69d712f157b1f7320d35568bb168dedf3e310b65e4de6ea6"} Jul 23 17:44:38.521009 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:38.520966 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-82.ec2.internal" podStartSLOduration=21.520953927 podStartE2EDuration="21.520953927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:44:21.478775457 +0000 UTC m=+5.557078810" watchObservedRunningTime="2026-07-23 17:44:38.520953927 +0000 UTC m=+22.599257279" Jul 23 17:44:38.521104 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:38.521035 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ggqsr" podStartSLOduration=3.07199174 podStartE2EDuration="22.52103022s" totalImagesPullingTime="19.44903848s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:44:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:44:38.520543083 +0000 UTC m=+22.598846436" watchObservedRunningTime="2026-07-23 17:44:38.52103022 +0000 UTC m=+22.599333573" Jul 23 17:44:38.539376 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:38.539336 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6nsmg" podStartSLOduration=3.114295929 podStartE2EDuration="22.539322115s" totalImagesPullingTime="19.425026186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:44:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:44:38.539307925 +0000 UTC m=+22.617611278" watchObservedRunningTime="2026-07-23 17:44:38.539322115 +0000 UTC m=+22.617625467" Jul 23 17:44:38.562345 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:38.562311 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-m8496" podStartSLOduration=3.147017656 podStartE2EDuration="22.562297338s" totalImagesPullingTime="19.415279682s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:44:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:44:38.562141921 +0000 UTC m=+22.640445274" watchObservedRunningTime="2026-07-23 17:44:38.562297338 +0000 UTC m=+22.640600690" Jul 23 17:44:38.650457 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:38.650414 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5xjz4" podStartSLOduration=3.189323855 podStartE2EDuration="22.650398792s" totalImagesPullingTime="19.461074937s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:44:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:44:38.625867961 +0000 UTC m=+22.704171314" watchObservedRunningTime="2026-07-23 17:44:38.650398792 +0000 UTC m=+22.728702191" Jul 23 17:44:38.650826 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:38.650803 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kcd4h" podStartSLOduration=3.216914864 podStartE2EDuration="22.650794404s" totalImagesPullingTime="19.43387954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:44:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:44:38.650357267 +0000 UTC m=+22.728660621" watchObservedRunningTime="2026-07-23 17:44:38.650794404 +0000 UTC m=+22.729097757" Jul 23 17:44:39.511361 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:39.511275 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rtmx5" event={"ID":"b1112f49-7d09-4c67-b66d-29e03194d1f9","Type":"ContainerStarted","Data":"de0d9c32707b6e1c575e324b68372dfa612269bd079146d7a0a328478dc91a18"} Jul 23 17:44:39.514263 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:39.514240 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" event={"ID":"84ce7a4f-1e92-434b-b939-616cce7d9188","Type":"ContainerStarted","Data":"8ed235c75b7ba12cb08ddac904d9cbc252f5fbdb870bd9834454b0b459797426"} Jul 23 17:44:39.514371 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:39.514299 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" event={"ID":"84ce7a4f-1e92-434b-b939-616cce7d9188","Type":"ContainerStarted","Data":"971841af47b5068612c3d18b48d1d791c6381723842138514b3fe10421c5c1c1"} Jul 23 17:44:39.514371 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:39.514327 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" event={"ID":"84ce7a4f-1e92-434b-b939-616cce7d9188","Type":"ContainerStarted","Data":"80dff81d01d4d2e298a9e0a033da829199c0fdd6ee302e88a74cc44ddf0958a7"} Jul 23 17:44:39.886761 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:39.886736 2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 23 17:44:40.358381 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:40.358281 2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T17:44:39.886755203Z","UUID":"ea1b0ad0-2c00-496e-8e0f-38b92e7988f3","Handler":null,"Name":"","Endpoint":""} Jul 23 17:44:40.360200 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:40.360179 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 23 17:44:40.360333 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:40.360233 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 23 17:44:40.399544 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:40.399516 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slcq6" Jul 23 17:44:40.399724 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:40.399655 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:44:40.399724 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:40.399661 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slcq6" podUID="cea169f8-ef99-4ce1-9559-029a6507402f" Jul 23 17:44:40.399847 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:40.399658 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:40.399847 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:40.399759 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2hlf" podUID="d7027def-1a58-4d31-a9ce-ea9814a550a3" Jul 23 17:44:40.399847 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:40.399838 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2j6vd" podUID="39d55095-246a-4ecf-bb04-36a9a8c92da4" Jul 23 17:44:40.518865 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:40.518824 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk8n6" event={"ID":"98b7a47e-3e44-401b-8d69-0052ef88824d","Type":"ContainerStarted","Data":"61faa2649fe2d2496476c668e17120924119811fc0703f1c9f2b62e48d14eb46"} Jul 23 17:44:41.032422 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:41.032380 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kcd4h" Jul 23 17:44:41.033166 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:41.033132 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kcd4h" Jul 23 17:44:41.050484 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:41.050422 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rtmx5" podStartSLOduration=5.65856161 podStartE2EDuration="25.050402306s" totalImagesPullingTime="19.391840696s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:44:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:44:39.534107552 +0000 UTC m=+23.612410904" watchObservedRunningTime="2026-07-23 17:44:41.050402306 +0000 UTC m=+25.128705656" Jul 23 17:44:41.526225 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:41.526187 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" event={"ID":"84ce7a4f-1e92-434b-b939-616cce7d9188","Type":"ContainerStarted","Data":"b81adaaec6e5aeeb6d6f77ae4526ae5dcbfe8fd351a969b4b7f75bd82592fea1"} Jul 23 17:44:42.399902 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:42.399715 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:44:42.400095 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:42.399715 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:42.400095 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:42.400006 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2hlf" podUID="d7027def-1a58-4d31-a9ce-ea9814a550a3" Jul 23 17:44:42.400198 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:42.400110 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2j6vd" podUID="39d55095-246a-4ecf-bb04-36a9a8c92da4" Jul 23 17:44:42.400198 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:42.399749 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slcq6" Jul 23 17:44:42.400270 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:42.400231 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slcq6" podUID="cea169f8-ef99-4ce1-9559-029a6507402f" Jul 23 17:44:42.515128 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:42.515093 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kcd4h" Jul 23 17:44:42.515867 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:42.515840 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kcd4h" Jul 23 17:44:42.530455 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:42.530419 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk8n6" event={"ID":"98b7a47e-3e44-401b-8d69-0052ef88824d","Type":"ContainerStarted","Data":"7b9bab2dfce206d4976c97284beafcd6566006310254740d080e2e1b940130f8"} Jul 23 17:44:42.555047 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:42.554981 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk8n6" podStartSLOduration=4.038174538 podStartE2EDuration="26.5549624s" totalImagesPullingTime="22.516787862s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:44:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:44:42.554580443 +0000 UTC m=+26.632883795" watchObservedRunningTime="2026-07-23 17:44:42.5549624 +0000 UTC m=+26.633265756" Jul 23 17:44:44.400445 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:44.400187 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:44:44.401209 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:44.400194 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:44.401209 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:44.400194 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slcq6" Jul 23 17:44:44.401209 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:44.400644 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2hlf" podUID="d7027def-1a58-4d31-a9ce-ea9814a550a3" Jul 23 17:44:44.401209 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:44.400749 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2j6vd" podUID="39d55095-246a-4ecf-bb04-36a9a8c92da4" Jul 23 17:44:44.401209 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:44.400798 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slcq6" podUID="cea169f8-ef99-4ce1-9559-029a6507402f" Jul 23 17:44:44.535614 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:44.535563 2565 generic.go:356] "Generic (PLEG): container finished" podID="5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce" containerID="c1cca9559c6e7ba8cf7c7b872ca6351e400f854b72b5e20523c5532f2b5b55ab" exitCode=0 Jul 23 17:44:44.535817 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:44.535634 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcfvv" event={"ID":"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce","Type":"ContainerDied","Data":"c1cca9559c6e7ba8cf7c7b872ca6351e400f854b72b5e20523c5532f2b5b55ab"} Jul 23 17:44:44.539372 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:44.539351 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" event={"ID":"84ce7a4f-1e92-434b-b939-616cce7d9188","Type":"ContainerStarted","Data":"b9ad287d708c2c2d95ef522c6b3fe82ad2fd4980baa4121db9b01bcbdf5eec33"} Jul 23 17:44:44.539631 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:44.539616 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:44.555226 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:44.555197 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:44.598007 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:44.597959 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" podStartSLOduration=9.094185072 podStartE2EDuration="28.597942253s" totalImagesPullingTime="19.503757181s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:44:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:44:44.596949504 +0000 UTC m=+28.675252858" watchObservedRunningTime="2026-07-23 17:44:44.597942253 +0000 UTC m=+28.676245600" Jul 23 17:44:45.542361 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:45.542321 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:45.542846 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:45.542392 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:45.557959 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:45.557933 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:44:45.836641 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:45.836158 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-slcq6"] Jul 23 17:44:45.836641 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:45.836334 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slcq6" Jul 23 17:44:45.836641 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:45.836482 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slcq6" podUID="cea169f8-ef99-4ce1-9559-029a6507402f" Jul 23 17:44:45.838735 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:45.838708 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2j6vd"] Jul 23 17:44:45.838895 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:45.838790 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:45.838945 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:45.838904 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2j6vd" podUID="39d55095-246a-4ecf-bb04-36a9a8c92da4" Jul 23 17:44:45.844525 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:45.844474 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t2hlf"] Jul 23 17:44:45.844615 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:45.844581 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:44:45.844734 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:45.844709 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2hlf" podUID="d7027def-1a58-4d31-a9ce-ea9814a550a3" Jul 23 17:44:46.545775 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:46.545735 2565 generic.go:356] "Generic (PLEG): container finished" podID="5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce" containerID="dd03861c7153d8bcc3e6c739f83804b0a625eb7e14dd834f8cb091122ad48882" exitCode=0 Jul 23 17:44:46.546140 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:46.545808 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcfvv" event={"ID":"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce","Type":"ContainerDied","Data":"dd03861c7153d8bcc3e6c739f83804b0a625eb7e14dd834f8cb091122ad48882"} Jul 23 17:44:47.399703 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:47.399636 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:44:47.399861 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:47.399637 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:47.399861 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:47.399762 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2hlf" podUID="d7027def-1a58-4d31-a9ce-ea9814a550a3" Jul 23 17:44:47.399861 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:47.399636 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slcq6" Jul 23 17:44:47.399861 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:47.399807 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2j6vd" podUID="39d55095-246a-4ecf-bb04-36a9a8c92da4" Jul 23 17:44:47.400055 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:47.399878 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slcq6" podUID="cea169f8-ef99-4ce1-9559-029a6507402f" Jul 23 17:44:48.552982 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:48.552811 2565 generic.go:356] "Generic (PLEG): container finished" podID="5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce" containerID="5de86cdf716ef93f13a3dad1ceeae96d91466dab9475c5b5dc21ce96b2047373" exitCode=0 Jul 23 17:44:48.552982 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:48.552894 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcfvv" event={"ID":"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce","Type":"ContainerDied","Data":"5de86cdf716ef93f13a3dad1ceeae96d91466dab9475c5b5dc21ce96b2047373"} Jul 23 17:44:49.399980 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:49.399945 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slcq6" Jul 23 17:44:49.399980 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:49.399963 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:44:49.399980 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:49.399989 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:49.400227 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:49.400075 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slcq6" podUID="cea169f8-ef99-4ce1-9559-029a6507402f" Jul 23 17:44:49.400227 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:49.400213 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2hlf" podUID="d7027def-1a58-4d31-a9ce-ea9814a550a3" Jul 23 17:44:49.400347 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:49.400295 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2j6vd" podUID="39d55095-246a-4ecf-bb04-36a9a8c92da4" Jul 23 17:44:50.157199 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.157157 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39d55095-246a-4ecf-bb04-36a9a8c92da4-original-pull-secret\") pod \"global-pull-secret-syncer-2j6vd\" (UID: \"39d55095-246a-4ecf-bb04-36a9a8c92da4\") " pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:50.157677 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.157227 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7027def-1a58-4d31-a9ce-ea9814a550a3-metrics-certs\") pod \"network-metrics-daemon-t2hlf\" (UID: \"d7027def-1a58-4d31-a9ce-ea9814a550a3\") " pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:44:50.157677 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.157259 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqwqs\" (UniqueName: \"kubernetes.io/projected/cea169f8-ef99-4ce1-9559-029a6507402f-kube-api-access-dqwqs\") pod \"network-check-target-slcq6\" (UID: \"cea169f8-ef99-4ce1-9559-029a6507402f\") " pod="openshift-network-diagnostics/network-check-target-slcq6" Jul 23 17:44:50.157677 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:50.157321 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 17:44:50.157677 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:50.157388 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d55095-246a-4ecf-bb04-36a9a8c92da4-original-pull-secret podName:39d55095-246a-4ecf-bb04-36a9a8c92da4 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:22.157370185 +0000 UTC m=+66.235673535 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39d55095-246a-4ecf-bb04-36a9a8c92da4-original-pull-secret") pod "global-pull-secret-syncer-2j6vd" (UID: "39d55095-246a-4ecf-bb04-36a9a8c92da4") : object "kube-system"/"original-pull-secret" not registered Jul 23 17:44:50.157677 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:50.157424 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:44:50.157677 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:50.157474 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 17:44:50.157677 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:50.157490 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7027def-1a58-4d31-a9ce-ea9814a550a3-metrics-certs podName:d7027def-1a58-4d31-a9ce-ea9814a550a3 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:22.157471641 +0000 UTC m=+66.235774984 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7027def-1a58-4d31-a9ce-ea9814a550a3-metrics-certs") pod "network-metrics-daemon-t2hlf" (UID: "d7027def-1a58-4d31-a9ce-ea9814a550a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 17:44:50.157677 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:50.157494 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 17:44:50.157677 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:50.157510 2565 projected.go:196] Error preparing data for projected volume kube-api-access-dqwqs for pod openshift-network-diagnostics/network-check-target-slcq6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:44:50.157677 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:50.157559 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cea169f8-ef99-4ce1-9559-029a6507402f-kube-api-access-dqwqs podName:cea169f8-ef99-4ce1-9559-029a6507402f nodeName:}" failed. No retries permitted until 2026-07-23 17:45:22.157545141 +0000 UTC m=+66.235848477 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dqwqs" (UniqueName: "kubernetes.io/projected/cea169f8-ef99-4ce1-9559-029a6507402f-kube-api-access-dqwqs") pod "network-check-target-slcq6" (UID: "cea169f8-ef99-4ce1-9559-029a6507402f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 17:44:50.743608 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.743567 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-82.ec2.internal" event="NodeReady" Jul 23 17:44:50.743762 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.743724 2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 23 17:44:50.782815 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.782789 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-dcf9785b6-pbsjf"] Jul 23 17:44:50.784511 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.784497 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:44:50.789070 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.789041 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 23 17:44:50.789208 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.789180 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 23 17:44:50.789270 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.789215 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s7c84\"" Jul 23 17:44:50.789412 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.789392 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 23 17:44:50.794347 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.794318 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 23 17:44:50.795753 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.795730 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vsw5b"] Jul 23 17:44:50.797414 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.797398 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vsw5b" Jul 23 17:44:50.799967 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.799943 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-dcf9785b6-pbsjf"] Jul 23 17:44:50.800148 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.800131 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 23 17:44:50.800216 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.800158 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 23 17:44:50.800316 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.800296 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bjl7v\"" Jul 23 17:44:50.800651 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.800634 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sp8s4"] Jul 23 17:44:50.802911 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.802893 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sp8s4" Jul 23 17:44:50.805224 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.805207 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 23 17:44:50.805224 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.805205 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xf6nc\"" Jul 23 17:44:50.805373 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.805205 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 23 17:44:50.805513 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.805498 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 23 17:44:50.812547 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.812529 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vsw5b"] Jul 23 17:44:50.826176 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.826118 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sp8s4"] Jul 23 17:44:50.861863 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.861825 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-tls\") pod \"image-registry-dcf9785b6-pbsjf\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:44:50.862032 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.861881 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-bound-sa-token\") pod \"image-registry-dcf9785b6-pbsjf\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:44:50.862032 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.861988 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0aaf3783-02d1-4b26-a61d-b960d83d1b30-ca-trust-extracted\") pod \"image-registry-dcf9785b6-pbsjf\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:44:50.862146 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.862045 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0aaf3783-02d1-4b26-a61d-b960d83d1b30-image-registry-private-configuration\") pod \"image-registry-dcf9785b6-pbsjf\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:44:50.862146 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.862082 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-certificates\") pod \"image-registry-dcf9785b6-pbsjf\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:44:50.862146 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.862107 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0aaf3783-02d1-4b26-a61d-b960d83d1b30-installation-pull-secrets\") pod \"image-registry-dcf9785b6-pbsjf\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:44:50.862146 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.862139 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn8d9\" (UniqueName: \"kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-kube-api-access-pn8d9\") pod \"image-registry-dcf9785b6-pbsjf\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:44:50.862297 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.862200 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0aaf3783-02d1-4b26-a61d-b960d83d1b30-trusted-ca\") pod \"image-registry-dcf9785b6-pbsjf\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:44:50.963197 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.963161 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0aaf3783-02d1-4b26-a61d-b960d83d1b30-ca-trust-extracted\") pod \"image-registry-dcf9785b6-pbsjf\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:44:50.963375 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.963226 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0aaf3783-02d1-4b26-a61d-b960d83d1b30-image-registry-private-configuration\") pod \"image-registry-dcf9785b6-pbsjf\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:44:50.963375 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.963263 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-certificates\") pod \"image-registry-dcf9785b6-pbsjf\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:44:50.963375 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.963287 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0aaf3783-02d1-4b26-a61d-b960d83d1b30-installation-pull-secrets\") pod \"image-registry-dcf9785b6-pbsjf\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:44:50.963375 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.963316 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn8d9\" (UniqueName: \"kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-kube-api-access-pn8d9\") pod \"image-registry-dcf9785b6-pbsjf\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:44:50.963585 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.963380 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-metrics-tls\") pod \"dns-default-vsw5b\" (UID: \"35c70c4c-f5f0-49a6-a61f-f8676deeb25a\") " pod="openshift-dns/dns-default-vsw5b" Jul 23 17:44:50.963585 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.963420 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0aaf3783-02d1-4b26-a61d-b960d83d1b30-trusted-ca\") pod \"image-registry-dcf9785b6-pbsjf\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:44:50.963585 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.963449 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-config-volume\") pod \"dns-default-vsw5b\" (UID: \"35c70c4c-f5f0-49a6-a61f-f8676deeb25a\") " pod="openshift-dns/dns-default-vsw5b" Jul 23 17:44:50.963585 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.963489 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-tls\") pod \"image-registry-dcf9785b6-pbsjf\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:44:50.963585 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.963511 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-tmp-dir\") pod \"dns-default-vsw5b\" (UID: \"35c70c4c-f5f0-49a6-a61f-f8676deeb25a\") " pod="openshift-dns/dns-default-vsw5b" Jul 23 17:44:50.963585 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.963533 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d532709-e70c-4eaf-b4e9-e650237862ea-cert\") pod \"ingress-canary-sp8s4\" (UID: \"5d532709-e70c-4eaf-b4e9-e650237862ea\") " pod="openshift-ingress-canary/ingress-canary-sp8s4" Jul 23 17:44:50.963585 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.963573 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-bound-sa-token\") pod \"image-registry-dcf9785b6-pbsjf\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:44:50.963945 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.963618 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv54f\" (UniqueName: \"kubernetes.io/projected/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-kube-api-access-jv54f\") pod \"dns-default-vsw5b\" (UID: \"35c70c4c-f5f0-49a6-a61f-f8676deeb25a\") " pod="openshift-dns/dns-default-vsw5b" Jul 23 17:44:50.963945 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.963672 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g864g\" (UniqueName: \"kubernetes.io/projected/5d532709-e70c-4eaf-b4e9-e650237862ea-kube-api-access-g864g\") pod \"ingress-canary-sp8s4\" (UID: \"5d532709-e70c-4eaf-b4e9-e650237862ea\") " pod="openshift-ingress-canary/ingress-canary-sp8s4" Jul 23 17:44:50.963945 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.963682 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0aaf3783-02d1-4b26-a61d-b960d83d1b30-ca-trust-extracted\") pod \"image-registry-dcf9785b6-pbsjf\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:44:50.963945 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:50.963782 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 17:44:50.963945 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:50.963799 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dcf9785b6-pbsjf: secret "image-registry-tls" not found Jul 23 17:44:50.963945 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:50.963860 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-tls podName:0aaf3783-02d1-4b26-a61d-b960d83d1b30 nodeName:}" failed. No retries permitted until 2026-07-23 17:44:51.463841306 +0000 UTC m=+35.542144650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-tls") pod "image-registry-dcf9785b6-pbsjf" (UID: "0aaf3783-02d1-4b26-a61d-b960d83d1b30") : secret "image-registry-tls" not found Jul 23 17:44:50.964252 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.963967 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-certificates\") pod \"image-registry-dcf9785b6-pbsjf\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:44:50.965376 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.965357 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0aaf3783-02d1-4b26-a61d-b960d83d1b30-trusted-ca\") pod \"image-registry-dcf9785b6-pbsjf\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:44:50.967791 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.967769 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0aaf3783-02d1-4b26-a61d-b960d83d1b30-image-registry-private-configuration\") pod \"image-registry-dcf9785b6-pbsjf\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:44:50.967898 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.967775 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0aaf3783-02d1-4b26-a61d-b960d83d1b30-installation-pull-secrets\") pod \"image-registry-dcf9785b6-pbsjf\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:44:50.972487 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.972437 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-bound-sa-token\") pod \"image-registry-dcf9785b6-pbsjf\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:44:50.972656 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:50.972508 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn8d9\" (UniqueName: \"kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-kube-api-access-pn8d9\") pod \"image-registry-dcf9785b6-pbsjf\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:44:51.064714 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:51.064684 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-config-volume\") pod \"dns-default-vsw5b\" (UID: \"35c70c4c-f5f0-49a6-a61f-f8676deeb25a\") " pod="openshift-dns/dns-default-vsw5b" Jul 23 17:44:51.064881 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:51.064738 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-tmp-dir\") pod \"dns-default-vsw5b\" (UID: \"35c70c4c-f5f0-49a6-a61f-f8676deeb25a\") " pod="openshift-dns/dns-default-vsw5b" Jul 23 17:44:51.064881 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:51.064763 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d532709-e70c-4eaf-b4e9-e650237862ea-cert\") pod \"ingress-canary-sp8s4\" (UID: \"5d532709-e70c-4eaf-b4e9-e650237862ea\") " pod="openshift-ingress-canary/ingress-canary-sp8s4" Jul 23 17:44:51.064881 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:51.064797 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv54f\" (UniqueName: \"kubernetes.io/projected/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-kube-api-access-jv54f\") pod \"dns-default-vsw5b\" (UID: \"35c70c4c-f5f0-49a6-a61f-f8676deeb25a\") " pod="openshift-dns/dns-default-vsw5b" Jul 23 17:44:51.064881 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:51.064837 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g864g\" (UniqueName: \"kubernetes.io/projected/5d532709-e70c-4eaf-b4e9-e650237862ea-kube-api-access-g864g\") pod \"ingress-canary-sp8s4\" (UID: \"5d532709-e70c-4eaf-b4e9-e650237862ea\") " pod="openshift-ingress-canary/ingress-canary-sp8s4" Jul 23 17:44:51.065091 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:51.064911 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-metrics-tls\") pod \"dns-default-vsw5b\" (UID: \"35c70c4c-f5f0-49a6-a61f-f8676deeb25a\") " pod="openshift-dns/dns-default-vsw5b" Jul 23 17:44:51.065091 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:51.065040 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 17:44:51.065187 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:51.065098 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-metrics-tls podName:35c70c4c-f5f0-49a6-a61f-f8676deeb25a nodeName:}" failed. No retries permitted until 2026-07-23 17:44:51.565079649 +0000 UTC m=+35.643382996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-metrics-tls") pod "dns-default-vsw5b" (UID: "35c70c4c-f5f0-49a6-a61f-f8676deeb25a") : secret "dns-default-metrics-tls" not found Jul 23 17:44:51.065187 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:51.065094 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-tmp-dir\") pod \"dns-default-vsw5b\" (UID: \"35c70c4c-f5f0-49a6-a61f-f8676deeb25a\") " pod="openshift-dns/dns-default-vsw5b" Jul 23 17:44:51.065438 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:51.065382 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-config-volume\") pod \"dns-default-vsw5b\" (UID: \"35c70c4c-f5f0-49a6-a61f-f8676deeb25a\") " pod="openshift-dns/dns-default-vsw5b" Jul 23 17:44:51.067444 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:51.067416 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d532709-e70c-4eaf-b4e9-e650237862ea-cert\") pod \"ingress-canary-sp8s4\" (UID: \"5d532709-e70c-4eaf-b4e9-e650237862ea\") " pod="openshift-ingress-canary/ingress-canary-sp8s4" Jul 23 17:44:51.073845 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:51.073822 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv54f\" (UniqueName: \"kubernetes.io/projected/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-kube-api-access-jv54f\") pod \"dns-default-vsw5b\" (UID: \"35c70c4c-f5f0-49a6-a61f-f8676deeb25a\") " pod="openshift-dns/dns-default-vsw5b" Jul 23 17:44:51.074085 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:51.074071 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g864g\" (UniqueName: \"kubernetes.io/projected/5d532709-e70c-4eaf-b4e9-e650237862ea-kube-api-access-g864g\") pod \"ingress-canary-sp8s4\" (UID: \"5d532709-e70c-4eaf-b4e9-e650237862ea\") " pod="openshift-ingress-canary/ingress-canary-sp8s4" Jul 23 17:44:51.112819 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:51.112738 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sp8s4" Jul 23 17:44:51.262250 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:51.262219 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sp8s4"] Jul 23 17:44:51.268697 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:44:51.268535 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d532709_e70c_4eaf_b4e9_e650237862ea.slice/crio-0a9c4795e2b89a1120326c9d8952fe43c0845c2a90aa467722d2db52b3ac0822 WatchSource:0}: Error finding container 0a9c4795e2b89a1120326c9d8952fe43c0845c2a90aa467722d2db52b3ac0822: Status 404 returned error can't find the container with id 0a9c4795e2b89a1120326c9d8952fe43c0845c2a90aa467722d2db52b3ac0822 Jul 23 17:44:51.400124 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:51.400009 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slcq6" Jul 23 17:44:51.400124 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:51.400020 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:44:51.400337 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:51.400164 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:44:51.403025 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:51.402996 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 17:44:51.403161 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:51.403129 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 17:44:51.403161 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:51.403144 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 17:44:51.403269 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:51.403144 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2xnzn\"" Jul 23 17:44:51.403269 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:51.403224 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mcdkw\"" Jul 23 17:44:51.403370 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:51.403294 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 17:44:51.468965 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:51.468929 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-tls\") pod \"image-registry-dcf9785b6-pbsjf\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:44:51.469143 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:51.469058 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 17:44:51.469143 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:51.469075 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dcf9785b6-pbsjf: secret "image-registry-tls" not found Jul 23 17:44:51.469250 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:51.469145 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-tls podName:0aaf3783-02d1-4b26-a61d-b960d83d1b30 nodeName:}" failed. No retries permitted until 2026-07-23 17:44:52.469125759 +0000 UTC m=+36.547429105 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-tls") pod "image-registry-dcf9785b6-pbsjf" (UID: "0aaf3783-02d1-4b26-a61d-b960d83d1b30") : secret "image-registry-tls" not found Jul 23 17:44:51.561172 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:51.561139 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sp8s4" event={"ID":"5d532709-e70c-4eaf-b4e9-e650237862ea","Type":"ContainerStarted","Data":"0a9c4795e2b89a1120326c9d8952fe43c0845c2a90aa467722d2db52b3ac0822"} Jul 23 17:44:51.569518 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:51.569493 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-metrics-tls\") pod \"dns-default-vsw5b\" (UID: \"35c70c4c-f5f0-49a6-a61f-f8676deeb25a\") " pod="openshift-dns/dns-default-vsw5b" Jul 23 17:44:51.569699 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:51.569672 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 17:44:51.569850 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:51.569743 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-metrics-tls podName:35c70c4c-f5f0-49a6-a61f-f8676deeb25a nodeName:}" failed. No retries permitted until 2026-07-23 17:44:52.569725316 +0000 UTC m=+36.648028650 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-metrics-tls") pod "dns-default-vsw5b" (UID: "35c70c4c-f5f0-49a6-a61f-f8676deeb25a") : secret "dns-default-metrics-tls" not found Jul 23 17:44:52.478511 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:52.478472 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-tls\") pod \"image-registry-dcf9785b6-pbsjf\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:44:52.478977 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:52.478684 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 17:44:52.478977 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:52.478711 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dcf9785b6-pbsjf: secret "image-registry-tls" not found Jul 23 17:44:52.478977 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:52.478780 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-tls podName:0aaf3783-02d1-4b26-a61d-b960d83d1b30 nodeName:}" failed. No retries permitted until 2026-07-23 17:44:54.478759835 +0000 UTC m=+38.557063186 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-tls") pod "image-registry-dcf9785b6-pbsjf" (UID: "0aaf3783-02d1-4b26-a61d-b960d83d1b30") : secret "image-registry-tls" not found Jul 23 17:44:52.579411 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:52.579373 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-metrics-tls\") pod \"dns-default-vsw5b\" (UID: \"35c70c4c-f5f0-49a6-a61f-f8676deeb25a\") " pod="openshift-dns/dns-default-vsw5b" Jul 23 17:44:52.579589 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:52.579545 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 17:44:52.579687 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:52.579652 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-metrics-tls podName:35c70c4c-f5f0-49a6-a61f-f8676deeb25a nodeName:}" failed. No retries permitted until 2026-07-23 17:44:54.579631397 +0000 UTC m=+38.657934738 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-metrics-tls") pod "dns-default-vsw5b" (UID: "35c70c4c-f5f0-49a6-a61f-f8676deeb25a") : secret "dns-default-metrics-tls" not found Jul 23 17:44:53.569965 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:53.569927 2565 generic.go:356] "Generic (PLEG): container finished" podID="5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce" containerID="3c300eee980f60f0a40f9cf80d2a021897ad97e846e573f81555563ed79cbd25" exitCode=0 Jul 23 17:44:53.570414 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:53.570008 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcfvv" event={"ID":"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce","Type":"ContainerDied","Data":"3c300eee980f60f0a40f9cf80d2a021897ad97e846e573f81555563ed79cbd25"} Jul 23 17:44:54.495327 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:54.495293 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-tls\") pod \"image-registry-dcf9785b6-pbsjf\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:44:54.495551 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:54.495420 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 17:44:54.495551 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:54.495434 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dcf9785b6-pbsjf: secret "image-registry-tls" not found Jul 23 17:44:54.495551 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:54.495505 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-tls podName:0aaf3783-02d1-4b26-a61d-b960d83d1b30 nodeName:}" failed. No retries permitted until 2026-07-23 17:44:58.495487704 +0000 UTC m=+42.573791035 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-tls") pod "image-registry-dcf9785b6-pbsjf" (UID: "0aaf3783-02d1-4b26-a61d-b960d83d1b30") : secret "image-registry-tls" not found Jul 23 17:44:54.576056 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:54.576021 2565 generic.go:356] "Generic (PLEG): container finished" podID="5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce" containerID="adca15a2fb9c1941e3bc7ea20d9d81703b1801dd738448711f8046b923f6a882" exitCode=0 Jul 23 17:44:54.576544 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:54.576115 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcfvv" event={"ID":"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce","Type":"ContainerDied","Data":"adca15a2fb9c1941e3bc7ea20d9d81703b1801dd738448711f8046b923f6a882"} Jul 23 17:44:54.596418 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:54.596388 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-metrics-tls\") pod \"dns-default-vsw5b\" (UID: \"35c70c4c-f5f0-49a6-a61f-f8676deeb25a\") " pod="openshift-dns/dns-default-vsw5b" Jul 23 17:44:54.596582 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:54.596565 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 17:44:54.596657 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:54.596649 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-metrics-tls podName:35c70c4c-f5f0-49a6-a61f-f8676deeb25a nodeName:}" failed. No retries permitted until 2026-07-23 17:44:58.596628823 +0000 UTC m=+42.674932171 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-metrics-tls") pod "dns-default-vsw5b" (UID: "35c70c4c-f5f0-49a6-a61f-f8676deeb25a") : secret "dns-default-metrics-tls" not found Jul 23 17:44:55.495796 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:55.495761 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kcd4h_4a324aac-61f5-46a9-b847-15730ba25434/konnectivity-agent/0.log" limit=1024 Jul 23 17:44:55.584671 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:55.584632 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcfvv" event={"ID":"5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce","Type":"ContainerStarted","Data":"5826615c7821d9fb05e6f94b6ed24c33762ee29478a0a1436f6740e65b4b8eaf"} Jul 23 17:44:55.586149 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:55.586125 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sp8s4" event={"ID":"5d532709-e70c-4eaf-b4e9-e650237862ea","Type":"ContainerStarted","Data":"aac964ecec0e79740a8c3e2a914daa59b3392fd62340d7ac5a631b558e831efd"} Jul 23 17:44:55.613008 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:55.612978 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kcd4h_4a324aac-61f5-46a9-b847-15730ba25434/konnectivity-agent/0.log" limit=1024 Jul 23 17:44:55.626883 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:55.626835 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vcfvv" podStartSLOduration=8.515710947 podStartE2EDuration="39.626818993s" totalImagesPullingTime="31.111108046s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:44:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:44:55.625364863 +0000 UTC m=+39.703668217" watchObservedRunningTime="2026-07-23 17:44:55.626818993 +0000 UTC m=+39.705122324" Jul 23 17:44:55.647766 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:55.647720 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sp8s4" podStartSLOduration=1.461699636 podStartE2EDuration="5.647704067s" totalImagesPullingTime="4.186004431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:44:55.646717263 +0000 UTC m=+39.725020626" watchObservedRunningTime="2026-07-23 17:44:55.647704067 +0000 UTC m=+39.726007420" Jul 23 17:44:55.849322 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:55.849284 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kcd4h_4a324aac-61f5-46a9-b847-15730ba25434/konnectivity-agent/0.log" limit=1024 Jul 23 17:44:55.920920 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:55.920889 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kcd4h_4a324aac-61f5-46a9-b847-15730ba25434/konnectivity-agent/0.log" limit=1024 Jul 23 17:44:55.993176 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:55.993146 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kcd4h_4a324aac-61f5-46a9-b847-15730ba25434/konnectivity-agent/0.log" limit=1024 Jul 23 17:44:58.529218 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:58.529174 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-tls\") pod \"image-registry-dcf9785b6-pbsjf\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:44:58.529629 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:58.529328 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 17:44:58.529629 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:58.529348 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dcf9785b6-pbsjf: secret "image-registry-tls" not found Jul 23 17:44:58.529629 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:58.529401 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-tls podName:0aaf3783-02d1-4b26-a61d-b960d83d1b30 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:06.529387147 +0000 UTC m=+50.607690478 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-tls") pod "image-registry-dcf9785b6-pbsjf" (UID: "0aaf3783-02d1-4b26-a61d-b960d83d1b30") : secret "image-registry-tls" not found Jul 23 17:44:58.630123 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:44:58.630080 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-metrics-tls\") pod \"dns-default-vsw5b\" (UID: \"35c70c4c-f5f0-49a6-a61f-f8676deeb25a\") " pod="openshift-dns/dns-default-vsw5b" Jul 23 17:44:58.630269 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:58.630199 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 17:44:58.630269 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:44:58.630250 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-metrics-tls podName:35c70c4c-f5f0-49a6-a61f-f8676deeb25a nodeName:}" failed. No retries permitted until 2026-07-23 17:45:06.630237398 +0000 UTC m=+50.708540729 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-metrics-tls") pod "dns-default-vsw5b" (UID: "35c70c4c-f5f0-49a6-a61f-f8676deeb25a") : secret "dns-default-metrics-tls" not found Jul 23 17:45:00.343586 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:00.343546 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kcd4h_4a324aac-61f5-46a9-b847-15730ba25434/konnectivity-agent/0.log" limit=1024 Jul 23 17:45:00.434187 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:00.434140 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kcd4h_4a324aac-61f5-46a9-b847-15730ba25434/konnectivity-agent/0.log" limit=1024 Jul 23 17:45:02.406068 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:02.406027 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kcd4h_4a324aac-61f5-46a9-b847-15730ba25434/konnectivity-agent/0.log" limit=1024 Jul 23 17:45:02.527327 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:02.527284 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kcd4h_4a324aac-61f5-46a9-b847-15730ba25434/konnectivity-agent/0.log" limit=1024 Jul 23 17:45:02.625279 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:02.625244 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kcd4h_4a324aac-61f5-46a9-b847-15730ba25434/konnectivity-agent/0.log" limit=1024 Jul 23 17:45:02.727098 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:02.727010 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kcd4h_4a324aac-61f5-46a9-b847-15730ba25434/konnectivity-agent/0.log" limit=1024 Jul 23 17:45:03.047861 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:03.047823 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kcd4h_4a324aac-61f5-46a9-b847-15730ba25434/konnectivity-agent/0.log" limit=1024 Jul 23 17:45:04.201743 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:04.201693 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kcd4h_4a324aac-61f5-46a9-b847-15730ba25434/konnectivity-agent/0.log" limit=1024 Jul 23 17:45:04.316950 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:04.316908 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kcd4h_4a324aac-61f5-46a9-b847-15730ba25434/konnectivity-agent/0.log" limit=1024 Jul 23 17:45:06.522526 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:06.522486 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kcd4h_4a324aac-61f5-46a9-b847-15730ba25434/konnectivity-agent/0.log" limit=1024 Jul 23 17:45:06.587494 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:06.587449 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-tls\") pod \"image-registry-dcf9785b6-pbsjf\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:45:06.587699 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:06.587631 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 17:45:06.587699 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:06.587651 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dcf9785b6-pbsjf: secret "image-registry-tls" not found Jul 23 17:45:06.587820 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:06.587758 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-tls podName:0aaf3783-02d1-4b26-a61d-b960d83d1b30 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:22.587736591 +0000 UTC m=+66.666039926 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-tls") pod "image-registry-dcf9785b6-pbsjf" (UID: "0aaf3783-02d1-4b26-a61d-b960d83d1b30") : secret "image-registry-tls" not found Jul 23 17:45:06.613644 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:06.613577 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kcd4h_4a324aac-61f5-46a9-b847-15730ba25434/konnectivity-agent/0.log" limit=1024 Jul 23 17:45:06.688707 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:06.688668 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-metrics-tls\") pod \"dns-default-vsw5b\" (UID: \"35c70c4c-f5f0-49a6-a61f-f8676deeb25a\") " pod="openshift-dns/dns-default-vsw5b" Jul 23 17:45:06.688876 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:06.688831 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 17:45:06.688939 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:06.688911 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-metrics-tls podName:35c70c4c-f5f0-49a6-a61f-f8676deeb25a nodeName:}" failed. No retries permitted until 2026-07-23 17:45:22.688890322 +0000 UTC m=+66.767193654 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-metrics-tls") pod "dns-default-vsw5b" (UID: "35c70c4c-f5f0-49a6-a61f-f8676deeb25a") : secret "dns-default-metrics-tls" not found Jul 23 17:45:06.712959 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:06.712927 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kcd4h_4a324aac-61f5-46a9-b847-15730ba25434/konnectivity-agent/0.log" limit=1024 Jul 23 17:45:06.816191 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:06.816094 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kcd4h_4a324aac-61f5-46a9-b847-15730ba25434/konnectivity-agent/0.log" limit=1024 Jul 23 17:45:06.914325 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:06.914288 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kcd4h_4a324aac-61f5-46a9-b847-15730ba25434/konnectivity-agent/0.log" limit=1024 Jul 23 17:45:17.509164 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.509132 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xh686"] Jul 23 17:45:17.545777 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.545750 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xh686"] Jul 23 17:45:17.545918 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.545843 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xh686" Jul 23 17:45:17.548654 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.548629 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 23 17:45:17.548900 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.548881 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-cw5zs\"" Jul 23 17:45:17.549022 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.548919 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 23 17:45:17.549022 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.548939 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 23 17:45:17.549262 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.549247 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 23 17:45:17.553062 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.553043 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nfqv\" (UniqueName: \"kubernetes.io/projected/e372e1b3-c8b6-4995-a770-6a970cd7f11e-kube-api-access-4nfqv\") pod \"kube-storage-version-migrator-operator-7658d9b749-xh686\" (UID: \"e372e1b3-c8b6-4995-a770-6a970cd7f11e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xh686" Jul 23 17:45:17.553168 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.553080 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e372e1b3-c8b6-4995-a770-6a970cd7f11e-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-xh686\" (UID: \"e372e1b3-c8b6-4995-a770-6a970cd7f11e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xh686" Jul 23 17:45:17.553168 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.553153 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e372e1b3-c8b6-4995-a770-6a970cd7f11e-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-xh686\" (UID: \"e372e1b3-c8b6-4995-a770-6a970cd7f11e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xh686" Jul 23 17:45:17.563882 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.563864 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pnj76" Jul 23 17:45:17.625668 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.625641 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-v5dk6"] Jul 23 17:45:17.648073 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.648051 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-v5dk6"] Jul 23 17:45:17.648223 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.648085 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dqmhz"] Jul 23 17:45:17.648223 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.648181 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-v5dk6" Jul 23 17:45:17.650829 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.650794 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 23 17:45:17.650829 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.650820 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 23 17:45:17.651008 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.650800 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-st8v4\"" Jul 23 17:45:17.651008 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.650800 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 23 17:45:17.653469 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.653443 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e372e1b3-c8b6-4995-a770-6a970cd7f11e-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-xh686\" (UID: \"e372e1b3-c8b6-4995-a770-6a970cd7f11e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xh686" Jul 23 17:45:17.653576 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.653522 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nfqv\" (UniqueName: \"kubernetes.io/projected/e372e1b3-c8b6-4995-a770-6a970cd7f11e-kube-api-access-4nfqv\") pod \"kube-storage-version-migrator-operator-7658d9b749-xh686\" (UID: \"e372e1b3-c8b6-4995-a770-6a970cd7f11e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xh686" Jul 23 17:45:17.653576 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.653569 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e372e1b3-c8b6-4995-a770-6a970cd7f11e-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-xh686\" (UID: \"e372e1b3-c8b6-4995-a770-6a970cd7f11e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xh686" Jul 23 17:45:17.654539 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.654341 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e372e1b3-c8b6-4995-a770-6a970cd7f11e-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-xh686\" (UID: \"e372e1b3-c8b6-4995-a770-6a970cd7f11e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xh686" Jul 23 17:45:17.657267 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.657245 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e372e1b3-c8b6-4995-a770-6a970cd7f11e-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-xh686\" (UID: \"e372e1b3-c8b6-4995-a770-6a970cd7f11e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xh686" Jul 23 17:45:17.664826 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.664800 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nfqv\" (UniqueName: \"kubernetes.io/projected/e372e1b3-c8b6-4995-a770-6a970cd7f11e-kube-api-access-4nfqv\") pod \"kube-storage-version-migrator-operator-7658d9b749-xh686\" (UID: \"e372e1b3-c8b6-4995-a770-6a970cd7f11e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xh686" Jul 23 17:45:17.668183 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.668160 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dqmhz"] Jul 23 17:45:17.668290 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.668263 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dqmhz" Jul 23 17:45:17.670739 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.670720 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-8f9z6\"" Jul 23 17:45:17.670816 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.670746 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 23 17:45:17.671291 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.671275 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 23 17:45:17.724741 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.724712 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-grgvm"] Jul 23 17:45:17.736773 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.736750 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-2mbb6"] Jul 23 17:45:17.736882 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.736867 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-grgvm" Jul 23 17:45:17.740549 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.740532 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-72xmt\"" Jul 23 17:45:17.747108 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.747088 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-xttxr"] Jul 23 17:45:17.747213 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.747197 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2mbb6" Jul 23 17:45:17.750480 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.750451 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 23 17:45:17.750961 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.750940 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-hn6z5\"" Jul 23 17:45:17.753839 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.753816 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 23 17:45:17.753968 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.753941 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmjhk\" (UniqueName: \"kubernetes.io/projected/daf01600-3fe9-4f23-b700-4c66740c4978-kube-api-access-tmjhk\") pod \"network-check-source-7bc5b6bf85-grgvm\" (UID: \"daf01600-3fe9-4f23-b700-4c66740c4978\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-grgvm" Jul 23 17:45:17.754075 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.754001 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtp45\" (UniqueName: \"kubernetes.io/projected/211a908c-6578-4515-a867-8da6b0125dea-kube-api-access-vtp45\") pod \"volume-data-source-validator-78cf4c6f6-dqmhz\" (UID: \"211a908c-6578-4515-a867-8da6b0125dea\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dqmhz" Jul 23 17:45:17.754075 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.754054 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dkt5\" (UniqueName: \"kubernetes.io/projected/c4e92771-7bcd-405d-b9f3-c054fec7f9cf-kube-api-access-2dkt5\") pod \"cluster-samples-operator-79b7c55f87-v5dk6\" (UID: \"c4e92771-7bcd-405d-b9f3-c054fec7f9cf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-v5dk6" Jul 23 17:45:17.754180 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.754123 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4e92771-7bcd-405d-b9f3-c054fec7f9cf-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-v5dk6\" (UID: \"c4e92771-7bcd-405d-b9f3-c054fec7f9cf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-v5dk6" Jul 23 17:45:17.754180 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.754151 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8b5828c5-b41b-4b6b-8300-7879fd076ff9-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-2mbb6\" (UID: \"8b5828c5-b41b-4b6b-8300-7879fd076ff9\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2mbb6" Jul 23 17:45:17.754180 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.754173 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8b5828c5-b41b-4b6b-8300-7879fd076ff9-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-2mbb6\" (UID: \"8b5828c5-b41b-4b6b-8300-7879fd076ff9\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2mbb6" Jul 23 17:45:17.764076 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.764058 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7b4c7f49cd-9kl9k"] Jul 23 17:45:17.764202 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.764186 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-xttxr" Jul 23 17:45:17.767163 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.767141 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 23 17:45:17.767266 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.767142 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 23 17:45:17.767266 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.767223 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-pb7l4\"" Jul 23 17:45:17.767385 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.767281 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 23 17:45:17.767566 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.767550 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 23 17:45:17.773375 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.773348 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kcd4h_4a324aac-61f5-46a9-b847-15730ba25434/konnectivity-agent/0.log" limit=1024 Jul 23 17:45:17.784067 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.784046 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-m7k4j"] Jul 23 17:45:17.784182 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.784164 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:17.787579 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.787563 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 23 17:45:17.787705 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.787651 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 23 17:45:17.787786 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.787720 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 23 17:45:17.787842 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.787785 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 23 17:45:17.787842 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.787792 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 23 17:45:17.787842 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.787795 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 23 17:45:17.788084 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.788070 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-hbfd6\"" Jul 23 17:45:17.806075 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.806055 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-grgvm"] Jul 23 17:45:17.806160 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.806085 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-2mbb6"] Jul 23 17:45:17.806160 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.806098 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-xttxr"] Jul 23 17:45:17.806160 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.806111 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-m7k4j"] Jul 23 17:45:17.806160 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.806128 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7b4c7f49cd-9kl9k"] Jul 23 17:45:17.806307 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.806191 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-m7k4j" Jul 23 17:45:17.810680 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.810662 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 23 17:45:17.811002 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.810973 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 23 17:45:17.811091 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.810982 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-f8vwv\"" Jul 23 17:45:17.811257 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.811240 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 23 17:45:17.811881 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.811861 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 23 17:45:17.816886 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.816868 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 23 17:45:17.827650 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.827630 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rwqpw"] Jul 23 17:45:17.845095 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.845071 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-n55cw"] Jul 23 17:45:17.845256 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.845239 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rwqpw" Jul 23 17:45:17.852878 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.852856 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 23 17:45:17.852983 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.852856 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 23 17:45:17.852983 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.852887 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 23 17:45:17.853273 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.853257 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xb5ff\"" Jul 23 17:45:17.854360 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.854345 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 23 17:45:17.854523 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.854504 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmjhk\" (UniqueName: \"kubernetes.io/projected/daf01600-3fe9-4f23-b700-4c66740c4978-kube-api-access-tmjhk\") pod \"network-check-source-7bc5b6bf85-grgvm\" (UID: \"daf01600-3fe9-4f23-b700-4c66740c4978\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-grgvm" Jul 23 17:45:17.854629 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.854539 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/13bc8906-58cc-4237-88d6-beade4de9408-default-certificate\") pod \"router-default-7b4c7f49cd-9kl9k\" (UID: \"13bc8906-58cc-4237-88d6-beade4de9408\") " pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:17.854629 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.854561 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8b5828c5-b41b-4b6b-8300-7879fd076ff9-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-2mbb6\" (UID: \"8b5828c5-b41b-4b6b-8300-7879fd076ff9\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2mbb6" Jul 23 17:45:17.854730 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.854621 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8b5828c5-b41b-4b6b-8300-7879fd076ff9-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-2mbb6\" (UID: \"8b5828c5-b41b-4b6b-8300-7879fd076ff9\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2mbb6" Jul 23 17:45:17.854730 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.854658 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1cca3236-5c90-4abb-9b9b-39e40b1903c3-trusted-ca\") pod \"console-operator-565b7b569d-m7k4j\" (UID: \"1cca3236-5c90-4abb-9b9b-39e40b1903c3\") " pod="openshift-console-operator/console-operator-565b7b569d-m7k4j" Jul 23 17:45:17.854730 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.854683 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/13bc8906-58cc-4237-88d6-beade4de9408-stats-auth\") pod \"router-default-7b4c7f49cd-9kl9k\" (UID: \"13bc8906-58cc-4237-88d6-beade4de9408\") " pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:17.854858 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.854737 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xh686" Jul 23 17:45:17.854858 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:17.854759 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 17:45:17.854943 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.854868 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtp45\" (UniqueName: \"kubernetes.io/projected/211a908c-6578-4515-a867-8da6b0125dea-kube-api-access-vtp45\") pod \"volume-data-source-validator-78cf4c6f6-dqmhz\" (UID: \"211a908c-6578-4515-a867-8da6b0125dea\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dqmhz" Jul 23 17:45:17.854943 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:17.854881 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b5828c5-b41b-4b6b-8300-7879fd076ff9-networking-console-plugin-cert podName:8b5828c5-b41b-4b6b-8300-7879fd076ff9 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:18.354860158 +0000 UTC m=+62.433163495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8b5828c5-b41b-4b6b-8300-7879fd076ff9-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-2mbb6" (UID: "8b5828c5-b41b-4b6b-8300-7879fd076ff9") : secret "networking-console-plugin-cert" not found Jul 23 17:45:17.854943 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.854919 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvhdk\" (UniqueName: \"kubernetes.io/projected/9b61f887-8f31-4935-9a55-626c4d3671e8-kube-api-access-nvhdk\") pod \"service-ca-operator-5bcbc94c78-xttxr\" (UID: \"9b61f887-8f31-4935-9a55-626c4d3671e8\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-xttxr" Jul 23 17:45:17.855071 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.854957 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1cca3236-5c90-4abb-9b9b-39e40b1903c3-serving-cert\") pod \"console-operator-565b7b569d-m7k4j\" (UID: \"1cca3236-5c90-4abb-9b9b-39e40b1903c3\") " pod="openshift-console-operator/console-operator-565b7b569d-m7k4j" Jul 23 17:45:17.855071 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.855008 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dkt5\" (UniqueName: \"kubernetes.io/projected/c4e92771-7bcd-405d-b9f3-c054fec7f9cf-kube-api-access-2dkt5\") pod \"cluster-samples-operator-79b7c55f87-v5dk6\" (UID: \"c4e92771-7bcd-405d-b9f3-c054fec7f9cf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-v5dk6" Jul 23 17:45:17.855071 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.855032 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e29fa838-5b80-4246-90f9-a9095e507279-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-rwqpw\" (UID: \"e29fa838-5b80-4246-90f9-a9095e507279\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rwqpw" Jul 23 17:45:17.855071 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.855051 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e29fa838-5b80-4246-90f9-a9095e507279-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-rwqpw\" (UID: \"e29fa838-5b80-4246-90f9-a9095e507279\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rwqpw" Jul 23 17:45:17.855212 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.855070 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9b61f887-8f31-4935-9a55-626c4d3671e8-config\") pod \"service-ca-operator-5bcbc94c78-xttxr\" (UID: \"9b61f887-8f31-4935-9a55-626c4d3671e8\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-xttxr" Jul 23 17:45:17.855212 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.855186 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4e92771-7bcd-405d-b9f3-c054fec7f9cf-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-v5dk6\" (UID: \"c4e92771-7bcd-405d-b9f3-c054fec7f9cf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-v5dk6" Jul 23 17:45:17.855212 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.855202 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8b5828c5-b41b-4b6b-8300-7879fd076ff9-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-2mbb6\" (UID: \"8b5828c5-b41b-4b6b-8300-7879fd076ff9\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2mbb6" Jul 23 17:45:17.855328 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.855217 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b61f887-8f31-4935-9a55-626c4d3671e8-serving-cert\") pod \"service-ca-operator-5bcbc94c78-xttxr\" (UID: \"9b61f887-8f31-4935-9a55-626c4d3671e8\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-xttxr" Jul 23 17:45:17.855328 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.855246 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl6xb\" (UniqueName: \"kubernetes.io/projected/13bc8906-58cc-4237-88d6-beade4de9408-kube-api-access-dl6xb\") pod \"router-default-7b4c7f49cd-9kl9k\" (UID: \"13bc8906-58cc-4237-88d6-beade4de9408\") " pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:17.855328 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:17.855281 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 17:45:17.855328 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.855288 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13bc8906-58cc-4237-88d6-beade4de9408-metrics-certs\") pod \"router-default-7b4c7f49cd-9kl9k\" (UID: \"13bc8906-58cc-4237-88d6-beade4de9408\") " pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:17.855328 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.855315 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13bc8906-58cc-4237-88d6-beade4de9408-service-ca-bundle\") pod \"router-default-7b4c7f49cd-9kl9k\" (UID: \"13bc8906-58cc-4237-88d6-beade4de9408\") " pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:17.855514 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:17.855335 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4e92771-7bcd-405d-b9f3-c054fec7f9cf-samples-operator-tls podName:c4e92771-7bcd-405d-b9f3-c054fec7f9cf nodeName:}" failed. No retries permitted until 2026-07-23 17:45:18.355315586 +0000 UTC m=+62.433618919 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c4e92771-7bcd-405d-b9f3-c054fec7f9cf-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-v5dk6" (UID: "c4e92771-7bcd-405d-b9f3-c054fec7f9cf") : secret "samples-operator-tls" not found Jul 23 17:45:17.855514 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.855367 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq47b\" (UniqueName: \"kubernetes.io/projected/e29fa838-5b80-4246-90f9-a9095e507279-kube-api-access-tq47b\") pod \"cluster-monitoring-operator-76cf8966f8-rwqpw\" (UID: \"e29fa838-5b80-4246-90f9-a9095e507279\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rwqpw" Jul 23 17:45:17.855514 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.855396 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1cca3236-5c90-4abb-9b9b-39e40b1903c3-config\") pod \"console-operator-565b7b569d-m7k4j\" (UID: \"1cca3236-5c90-4abb-9b9b-39e40b1903c3\") " pod="openshift-console-operator/console-operator-565b7b569d-m7k4j" Jul 23 17:45:17.855514 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.855421 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82jt9\" (UniqueName: \"kubernetes.io/projected/1cca3236-5c90-4abb-9b9b-39e40b1903c3-kube-api-access-82jt9\") pod \"console-operator-565b7b569d-m7k4j\" (UID: \"1cca3236-5c90-4abb-9b9b-39e40b1903c3\") " pod="openshift-console-operator/console-operator-565b7b569d-m7k4j" Jul 23 17:45:17.863193 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.863174 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rwqpw"] Jul 23 17:45:17.863273 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.863202 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-n55cw"] Jul 23 17:45:17.863306 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.863273 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-n55cw" Jul 23 17:45:17.866710 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.865960 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 23 17:45:17.866710 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.866207 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-65dtf\"" Jul 23 17:45:17.866710 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.866248 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 23 17:45:17.866710 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.866468 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 23 17:45:17.866710 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.866706 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 23 17:45:17.879836 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.879809 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 23 17:45:17.885688 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.885638 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dkt5\" (UniqueName: \"kubernetes.io/projected/c4e92771-7bcd-405d-b9f3-c054fec7f9cf-kube-api-access-2dkt5\") pod \"cluster-samples-operator-79b7c55f87-v5dk6\" (UID: \"c4e92771-7bcd-405d-b9f3-c054fec7f9cf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-v5dk6" Jul 23 17:45:17.886715 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.886694 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtp45\" (UniqueName: \"kubernetes.io/projected/211a908c-6578-4515-a867-8da6b0125dea-kube-api-access-vtp45\") pod \"volume-data-source-validator-78cf4c6f6-dqmhz\" (UID: \"211a908c-6578-4515-a867-8da6b0125dea\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dqmhz" Jul 23 17:45:17.902531 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.902503 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 17:45:17.914853 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.914826 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 17:45:17.925740 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.925710 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmjhk\" (UniqueName: \"kubernetes.io/projected/daf01600-3fe9-4f23-b700-4c66740c4978-kube-api-access-tmjhk\") pod \"network-check-source-7bc5b6bf85-grgvm\" (UID: \"daf01600-3fe9-4f23-b700-4c66740c4978\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-grgvm" Jul 23 17:45:17.956366 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.956341 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82jt9\" (UniqueName: \"kubernetes.io/projected/1cca3236-5c90-4abb-9b9b-39e40b1903c3-kube-api-access-82jt9\") pod \"console-operator-565b7b569d-m7k4j\" (UID: \"1cca3236-5c90-4abb-9b9b-39e40b1903c3\") " pod="openshift-console-operator/console-operator-565b7b569d-m7k4j" Jul 23 17:45:17.956511 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.956402 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mpsh\" (UniqueName: \"kubernetes.io/projected/f590b7bd-8791-41a3-a520-9ba3559d3371-kube-api-access-9mpsh\") pod \"insights-operator-555d9d7885-n55cw\" (UID: \"f590b7bd-8791-41a3-a520-9ba3559d3371\") " pod="openshift-insights/insights-operator-555d9d7885-n55cw" Jul 23 17:45:17.956586 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.956538 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f590b7bd-8791-41a3-a520-9ba3559d3371-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-n55cw\" (UID: \"f590b7bd-8791-41a3-a520-9ba3559d3371\") " pod="openshift-insights/insights-operator-555d9d7885-n55cw" Jul 23 17:45:17.956672 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.956617 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/13bc8906-58cc-4237-88d6-beade4de9408-default-certificate\") pod \"router-default-7b4c7f49cd-9kl9k\" (UID: \"13bc8906-58cc-4237-88d6-beade4de9408\") " pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:17.956723 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.956669 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1cca3236-5c90-4abb-9b9b-39e40b1903c3-trusted-ca\") pod \"console-operator-565b7b569d-m7k4j\" (UID: \"1cca3236-5c90-4abb-9b9b-39e40b1903c3\") " pod="openshift-console-operator/console-operator-565b7b569d-m7k4j" Jul 23 17:45:17.956723 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.956694 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/13bc8906-58cc-4237-88d6-beade4de9408-stats-auth\") pod \"router-default-7b4c7f49cd-9kl9k\" (UID: \"13bc8906-58cc-4237-88d6-beade4de9408\") " pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:17.956817 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.956747 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvhdk\" (UniqueName: \"kubernetes.io/projected/9b61f887-8f31-4935-9a55-626c4d3671e8-kube-api-access-nvhdk\") pod \"service-ca-operator-5bcbc94c78-xttxr\" (UID: \"9b61f887-8f31-4935-9a55-626c4d3671e8\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-xttxr" Jul 23 17:45:17.956817 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.956778 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1cca3236-5c90-4abb-9b9b-39e40b1903c3-serving-cert\") pod \"console-operator-565b7b569d-m7k4j\" (UID: \"1cca3236-5c90-4abb-9b9b-39e40b1903c3\") " pod="openshift-console-operator/console-operator-565b7b569d-m7k4j" Jul 23 17:45:17.956916 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.956815 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e29fa838-5b80-4246-90f9-a9095e507279-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-rwqpw\" (UID: \"e29fa838-5b80-4246-90f9-a9095e507279\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rwqpw" Jul 23 17:45:17.956916 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.956839 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f590b7bd-8791-41a3-a520-9ba3559d3371-tmp\") pod \"insights-operator-555d9d7885-n55cw\" (UID: \"f590b7bd-8791-41a3-a520-9ba3559d3371\") " pod="openshift-insights/insights-operator-555d9d7885-n55cw" Jul 23 17:45:17.956916 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.956866 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e29fa838-5b80-4246-90f9-a9095e507279-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-rwqpw\" (UID: \"e29fa838-5b80-4246-90f9-a9095e507279\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rwqpw" Jul 23 17:45:17.956916 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.956899 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9b61f887-8f31-4935-9a55-626c4d3671e8-config\") pod \"service-ca-operator-5bcbc94c78-xttxr\" (UID: \"9b61f887-8f31-4935-9a55-626c4d3671e8\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-xttxr" Jul 23 17:45:17.957098 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.956942 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b61f887-8f31-4935-9a55-626c4d3671e8-serving-cert\") pod \"service-ca-operator-5bcbc94c78-xttxr\" (UID: \"9b61f887-8f31-4935-9a55-626c4d3671e8\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-xttxr" Jul 23 17:45:17.957098 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.956971 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl6xb\" (UniqueName: \"kubernetes.io/projected/13bc8906-58cc-4237-88d6-beade4de9408-kube-api-access-dl6xb\") pod \"router-default-7b4c7f49cd-9kl9k\" (UID: \"13bc8906-58cc-4237-88d6-beade4de9408\") " pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:17.958323 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:17.957298 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 17:45:17.958323 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:17.957370 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e29fa838-5b80-4246-90f9-a9095e507279-cluster-monitoring-operator-tls podName:e29fa838-5b80-4246-90f9-a9095e507279 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:18.457353289 +0000 UTC m=+62.535656630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e29fa838-5b80-4246-90f9-a9095e507279-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-rwqpw" (UID: "e29fa838-5b80-4246-90f9-a9095e507279") : secret "cluster-monitoring-operator-tls" not found Jul 23 17:45:17.958323 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.957621 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1cca3236-5c90-4abb-9b9b-39e40b1903c3-trusted-ca\") pod \"console-operator-565b7b569d-m7k4j\" (UID: \"1cca3236-5c90-4abb-9b9b-39e40b1903c3\") " pod="openshift-console-operator/console-operator-565b7b569d-m7k4j" Jul 23 17:45:17.958323 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.957673 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e29fa838-5b80-4246-90f9-a9095e507279-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-rwqpw\" (UID: \"e29fa838-5b80-4246-90f9-a9095e507279\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rwqpw" Jul 23 17:45:17.958323 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.957721 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13bc8906-58cc-4237-88d6-beade4de9408-metrics-certs\") pod \"router-default-7b4c7f49cd-9kl9k\" (UID: \"13bc8906-58cc-4237-88d6-beade4de9408\") " pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:17.958323 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.957752 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f590b7bd-8791-41a3-a520-9ba3559d3371-service-ca-bundle\") pod \"insights-operator-555d9d7885-n55cw\" (UID: \"f590b7bd-8791-41a3-a520-9ba3559d3371\") " pod="openshift-insights/insights-operator-555d9d7885-n55cw" Jul 23 17:45:17.958323 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.957791 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13bc8906-58cc-4237-88d6-beade4de9408-service-ca-bundle\") pod \"router-default-7b4c7f49cd-9kl9k\" (UID: \"13bc8906-58cc-4237-88d6-beade4de9408\") " pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:17.958323 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.957816 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f590b7bd-8791-41a3-a520-9ba3559d3371-snapshots\") pod \"insights-operator-555d9d7885-n55cw\" (UID: \"f590b7bd-8791-41a3-a520-9ba3559d3371\") " pod="openshift-insights/insights-operator-555d9d7885-n55cw" Jul 23 17:45:17.958323 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.957846 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq47b\" (UniqueName: \"kubernetes.io/projected/e29fa838-5b80-4246-90f9-a9095e507279-kube-api-access-tq47b\") pod \"cluster-monitoring-operator-76cf8966f8-rwqpw\" (UID: \"e29fa838-5b80-4246-90f9-a9095e507279\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rwqpw" Jul 23 17:45:17.958323 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.957891 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1cca3236-5c90-4abb-9b9b-39e40b1903c3-config\") pod \"console-operator-565b7b569d-m7k4j\" (UID: \"1cca3236-5c90-4abb-9b9b-39e40b1903c3\") " pod="openshift-console-operator/console-operator-565b7b569d-m7k4j" Jul 23 17:45:17.958323 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.957913 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f590b7bd-8791-41a3-a520-9ba3559d3371-serving-cert\") pod \"insights-operator-555d9d7885-n55cw\" (UID: \"f590b7bd-8791-41a3-a520-9ba3559d3371\") " pod="openshift-insights/insights-operator-555d9d7885-n55cw" Jul 23 17:45:17.958323 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:17.958037 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 17:45:17.958323 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:17.958087 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13bc8906-58cc-4237-88d6-beade4de9408-metrics-certs podName:13bc8906-58cc-4237-88d6-beade4de9408 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:18.458072616 +0000 UTC m=+62.536375948 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13bc8906-58cc-4237-88d6-beade4de9408-metrics-certs") pod "router-default-7b4c7f49cd-9kl9k" (UID: "13bc8906-58cc-4237-88d6-beade4de9408") : secret "router-metrics-certs-default" not found Jul 23 17:45:17.958323 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:17.958171 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13bc8906-58cc-4237-88d6-beade4de9408-service-ca-bundle podName:13bc8906-58cc-4237-88d6-beade4de9408 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:18.458160569 +0000 UTC m=+62.536463899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/13bc8906-58cc-4237-88d6-beade4de9408-service-ca-bundle") pod "router-default-7b4c7f49cd-9kl9k" (UID: "13bc8906-58cc-4237-88d6-beade4de9408") : configmap references non-existent config key: service-ca.crt Jul 23 17:45:17.959143 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.958035 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9b61f887-8f31-4935-9a55-626c4d3671e8-config\") pod \"service-ca-operator-5bcbc94c78-xttxr\" (UID: \"9b61f887-8f31-4935-9a55-626c4d3671e8\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-xttxr" Jul 23 17:45:17.959829 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.959803 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1cca3236-5c90-4abb-9b9b-39e40b1903c3-config\") pod \"console-operator-565b7b569d-m7k4j\" (UID: \"1cca3236-5c90-4abb-9b9b-39e40b1903c3\") " pod="openshift-console-operator/console-operator-565b7b569d-m7k4j" Jul 23 17:45:17.960395 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.960344 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/13bc8906-58cc-4237-88d6-beade4de9408-default-certificate\") pod \"router-default-7b4c7f49cd-9kl9k\" (UID: \"13bc8906-58cc-4237-88d6-beade4de9408\") " pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:17.960992 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.960966 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/13bc8906-58cc-4237-88d6-beade4de9408-stats-auth\") pod \"router-default-7b4c7f49cd-9kl9k\" (UID: \"13bc8906-58cc-4237-88d6-beade4de9408\") " pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:17.961367 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.961341 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1cca3236-5c90-4abb-9b9b-39e40b1903c3-serving-cert\") pod \"console-operator-565b7b569d-m7k4j\" (UID: \"1cca3236-5c90-4abb-9b9b-39e40b1903c3\") " pod="openshift-console-operator/console-operator-565b7b569d-m7k4j" Jul 23 17:45:17.961824 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.961803 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b61f887-8f31-4935-9a55-626c4d3671e8-serving-cert\") pod \"service-ca-operator-5bcbc94c78-xttxr\" (UID: \"9b61f887-8f31-4935-9a55-626c4d3671e8\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-xttxr" Jul 23 17:45:17.965583 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.965561 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82jt9\" (UniqueName: \"kubernetes.io/projected/1cca3236-5c90-4abb-9b9b-39e40b1903c3-kube-api-access-82jt9\") pod \"console-operator-565b7b569d-m7k4j\" (UID: \"1cca3236-5c90-4abb-9b9b-39e40b1903c3\") " pod="openshift-console-operator/console-operator-565b7b569d-m7k4j" Jul 23 17:45:17.973772 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.973747 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl6xb\" (UniqueName: \"kubernetes.io/projected/13bc8906-58cc-4237-88d6-beade4de9408-kube-api-access-dl6xb\") pod \"router-default-7b4c7f49cd-9kl9k\" (UID: \"13bc8906-58cc-4237-88d6-beade4de9408\") " pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:17.974754 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.974735 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvhdk\" (UniqueName: \"kubernetes.io/projected/9b61f887-8f31-4935-9a55-626c4d3671e8-kube-api-access-nvhdk\") pod \"service-ca-operator-5bcbc94c78-xttxr\" (UID: \"9b61f887-8f31-4935-9a55-626c4d3671e8\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-xttxr" Jul 23 17:45:17.976636 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.976620 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dqmhz" Jul 23 17:45:17.979952 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:17.979931 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq47b\" (UniqueName: \"kubernetes.io/projected/e29fa838-5b80-4246-90f9-a9095e507279-kube-api-access-tq47b\") pod \"cluster-monitoring-operator-76cf8966f8-rwqpw\" (UID: \"e29fa838-5b80-4246-90f9-a9095e507279\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rwqpw" Jul 23 17:45:18.006472 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.006445 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kcd4h_4a324aac-61f5-46a9-b847-15730ba25434/konnectivity-agent/0.log" limit=1024 Jul 23 17:45:18.012269 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.012153 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xh686"] Jul 23 17:45:18.014828 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:45:18.014776 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode372e1b3_c8b6_4995_a770_6a970cd7f11e.slice/crio-56abbec210fc2c91e24134ac891abb7059f60e0db2626bba5889222692a5716b WatchSource:0}: Error finding container 56abbec210fc2c91e24134ac891abb7059f60e0db2626bba5889222692a5716b: Status 404 returned error can't find the container with id 56abbec210fc2c91e24134ac891abb7059f60e0db2626bba5889222692a5716b Jul 23 17:45:18.045569 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.045529 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-grgvm" Jul 23 17:45:18.059110 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.059084 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f590b7bd-8791-41a3-a520-9ba3559d3371-tmp\") pod \"insights-operator-555d9d7885-n55cw\" (UID: \"f590b7bd-8791-41a3-a520-9ba3559d3371\") " pod="openshift-insights/insights-operator-555d9d7885-n55cw" Jul 23 17:45:18.059227 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.059170 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f590b7bd-8791-41a3-a520-9ba3559d3371-service-ca-bundle\") pod \"insights-operator-555d9d7885-n55cw\" (UID: \"f590b7bd-8791-41a3-a520-9ba3559d3371\") " pod="openshift-insights/insights-operator-555d9d7885-n55cw" Jul 23 17:45:18.059227 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.059204 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f590b7bd-8791-41a3-a520-9ba3559d3371-snapshots\") pod \"insights-operator-555d9d7885-n55cw\" (UID: \"f590b7bd-8791-41a3-a520-9ba3559d3371\") " pod="openshift-insights/insights-operator-555d9d7885-n55cw" Jul 23 17:45:18.059312 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.059234 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f590b7bd-8791-41a3-a520-9ba3559d3371-serving-cert\") pod \"insights-operator-555d9d7885-n55cw\" (UID: \"f590b7bd-8791-41a3-a520-9ba3559d3371\") " pod="openshift-insights/insights-operator-555d9d7885-n55cw" Jul 23 17:45:18.059406 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.059387 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mpsh\" (UniqueName: \"kubernetes.io/projected/f590b7bd-8791-41a3-a520-9ba3559d3371-kube-api-access-9mpsh\") pod \"insights-operator-555d9d7885-n55cw\" (UID: \"f590b7bd-8791-41a3-a520-9ba3559d3371\") " pod="openshift-insights/insights-operator-555d9d7885-n55cw" Jul 23 17:45:18.059482 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.059439 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f590b7bd-8791-41a3-a520-9ba3559d3371-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-n55cw\" (UID: \"f590b7bd-8791-41a3-a520-9ba3559d3371\") " pod="openshift-insights/insights-operator-555d9d7885-n55cw" Jul 23 17:45:18.059555 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.059518 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f590b7bd-8791-41a3-a520-9ba3559d3371-tmp\") pod \"insights-operator-555d9d7885-n55cw\" (UID: \"f590b7bd-8791-41a3-a520-9ba3559d3371\") " pod="openshift-insights/insights-operator-555d9d7885-n55cw" Jul 23 17:45:18.059873 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.059851 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f590b7bd-8791-41a3-a520-9ba3559d3371-service-ca-bundle\") pod \"insights-operator-555d9d7885-n55cw\" (UID: \"f590b7bd-8791-41a3-a520-9ba3559d3371\") " pod="openshift-insights/insights-operator-555d9d7885-n55cw" Jul 23 17:45:18.059975 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.059861 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f590b7bd-8791-41a3-a520-9ba3559d3371-snapshots\") pod \"insights-operator-555d9d7885-n55cw\" (UID: \"f590b7bd-8791-41a3-a520-9ba3559d3371\") " pod="openshift-insights/insights-operator-555d9d7885-n55cw" Jul 23 17:45:18.060532 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.060508 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f590b7bd-8791-41a3-a520-9ba3559d3371-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-n55cw\" (UID: \"f590b7bd-8791-41a3-a520-9ba3559d3371\") " pod="openshift-insights/insights-operator-555d9d7885-n55cw" Jul 23 17:45:18.061860 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.061835 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f590b7bd-8791-41a3-a520-9ba3559d3371-serving-cert\") pod \"insights-operator-555d9d7885-n55cw\" (UID: \"f590b7bd-8791-41a3-a520-9ba3559d3371\") " pod="openshift-insights/insights-operator-555d9d7885-n55cw" Jul 23 17:45:18.069092 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.069071 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mpsh\" (UniqueName: \"kubernetes.io/projected/f590b7bd-8791-41a3-a520-9ba3559d3371-kube-api-access-9mpsh\") pod \"insights-operator-555d9d7885-n55cw\" (UID: \"f590b7bd-8791-41a3-a520-9ba3559d3371\") " pod="openshift-insights/insights-operator-555d9d7885-n55cw" Jul 23 17:45:18.074961 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.074912 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-xttxr" Jul 23 17:45:18.099005 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.098945 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kcd4h_4a324aac-61f5-46a9-b847-15730ba25434/konnectivity-agent/0.log" limit=1024 Jul 23 17:45:18.115768 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.115453 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-m7k4j" Jul 23 17:45:18.132299 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.132169 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dqmhz"] Jul 23 17:45:18.172681 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.172414 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-n55cw" Jul 23 17:45:18.220462 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.220226 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-grgvm"] Jul 23 17:45:18.258128 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.258029 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-xttxr"] Jul 23 17:45:18.261858 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:45:18.261804 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b61f887_8f31_4935_9a55_626c4d3671e8.slice/crio-7672c42e9f4ccf1ae84b259b5e4d5243d3836ac61cc6be849826a2df453e3af7 WatchSource:0}: Error finding container 7672c42e9f4ccf1ae84b259b5e4d5243d3836ac61cc6be849826a2df453e3af7: Status 404 returned error can't find the container with id 7672c42e9f4ccf1ae84b259b5e4d5243d3836ac61cc6be849826a2df453e3af7 Jul 23 17:45:18.297382 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.297352 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-m7k4j"] Jul 23 17:45:18.300738 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:45:18.300701 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cca3236_5c90_4abb_9b9b_39e40b1903c3.slice/crio-cd2a8dd21738805800c599ca2ee0215231e1bcb5d2f14f5e8c3be6970c69fbe9 WatchSource:0}: Error finding container cd2a8dd21738805800c599ca2ee0215231e1bcb5d2f14f5e8c3be6970c69fbe9: Status 404 returned error can't find the container with id cd2a8dd21738805800c599ca2ee0215231e1bcb5d2f14f5e8c3be6970c69fbe9 Jul 23 17:45:18.337788 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.337751 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-n55cw"] Jul 23 17:45:18.340808 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:45:18.340788 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf590b7bd_8791_41a3_a520_9ba3559d3371.slice/crio-52cbd7a54ed46b93b43b35eea498d90cc621a7fd5536fe1e2edf09307e11be21 WatchSource:0}: Error finding container 52cbd7a54ed46b93b43b35eea498d90cc621a7fd5536fe1e2edf09307e11be21: Status 404 returned error can't find the container with id 52cbd7a54ed46b93b43b35eea498d90cc621a7fd5536fe1e2edf09307e11be21 Jul 23 17:45:18.363301 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.363278 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8b5828c5-b41b-4b6b-8300-7879fd076ff9-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-2mbb6\" (UID: \"8b5828c5-b41b-4b6b-8300-7879fd076ff9\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2mbb6" Jul 23 17:45:18.363405 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.363380 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4e92771-7bcd-405d-b9f3-c054fec7f9cf-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-v5dk6\" (UID: \"c4e92771-7bcd-405d-b9f3-c054fec7f9cf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-v5dk6" Jul 23 17:45:18.363465 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:18.363425 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 17:45:18.363533 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:18.363482 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b5828c5-b41b-4b6b-8300-7879fd076ff9-networking-console-plugin-cert podName:8b5828c5-b41b-4b6b-8300-7879fd076ff9 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:19.363467182 +0000 UTC m=+63.441770514 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8b5828c5-b41b-4b6b-8300-7879fd076ff9-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-2mbb6" (UID: "8b5828c5-b41b-4b6b-8300-7879fd076ff9") : secret "networking-console-plugin-cert" not found Jul 23 17:45:18.363533 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:18.363516 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 17:45:18.363653 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:18.363572 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4e92771-7bcd-405d-b9f3-c054fec7f9cf-samples-operator-tls podName:c4e92771-7bcd-405d-b9f3-c054fec7f9cf nodeName:}" failed. No retries permitted until 2026-07-23 17:45:19.3635562 +0000 UTC m=+63.441859532 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c4e92771-7bcd-405d-b9f3-c054fec7f9cf-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-v5dk6" (UID: "c4e92771-7bcd-405d-b9f3-c054fec7f9cf") : secret "samples-operator-tls" not found Jul 23 17:45:18.464580 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.464548 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e29fa838-5b80-4246-90f9-a9095e507279-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-rwqpw\" (UID: \"e29fa838-5b80-4246-90f9-a9095e507279\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rwqpw" Jul 23 17:45:18.464742 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.464617 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13bc8906-58cc-4237-88d6-beade4de9408-metrics-certs\") pod \"router-default-7b4c7f49cd-9kl9k\" (UID: \"13bc8906-58cc-4237-88d6-beade4de9408\") " pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:18.464742 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.464644 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13bc8906-58cc-4237-88d6-beade4de9408-service-ca-bundle\") pod \"router-default-7b4c7f49cd-9kl9k\" (UID: \"13bc8906-58cc-4237-88d6-beade4de9408\") " pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:18.464742 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:18.464699 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 17:45:18.464839 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:18.464743 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 17:45:18.464839 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:18.464750 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13bc8906-58cc-4237-88d6-beade4de9408-service-ca-bundle podName:13bc8906-58cc-4237-88d6-beade4de9408 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:19.464737089 +0000 UTC m=+63.543040420 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/13bc8906-58cc-4237-88d6-beade4de9408-service-ca-bundle") pod "router-default-7b4c7f49cd-9kl9k" (UID: "13bc8906-58cc-4237-88d6-beade4de9408") : configmap references non-existent config key: service-ca.crt Jul 23 17:45:18.464839 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:18.464793 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e29fa838-5b80-4246-90f9-a9095e507279-cluster-monitoring-operator-tls podName:e29fa838-5b80-4246-90f9-a9095e507279 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:19.46477794 +0000 UTC m=+63.543081274 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e29fa838-5b80-4246-90f9-a9095e507279-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-rwqpw" (UID: "e29fa838-5b80-4246-90f9-a9095e507279") : secret "cluster-monitoring-operator-tls" not found Jul 23 17:45:18.464839 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:18.464809 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13bc8906-58cc-4237-88d6-beade4de9408-metrics-certs podName:13bc8906-58cc-4237-88d6-beade4de9408 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:19.464800053 +0000 UTC m=+63.543103384 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13bc8906-58cc-4237-88d6-beade4de9408-metrics-certs") pod "router-default-7b4c7f49cd-9kl9k" (UID: "13bc8906-58cc-4237-88d6-beade4de9408") : secret "router-metrics-certs-default" not found Jul 23 17:45:18.640094 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.639982 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dqmhz" event={"ID":"211a908c-6578-4515-a867-8da6b0125dea","Type":"ContainerStarted","Data":"c7ab481c0ec6665fc002d77bbfdeeb00bcaad83e7bd13af69141f94ba251929d"} Jul 23 17:45:18.641284 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.641258 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xh686" event={"ID":"e372e1b3-c8b6-4995-a770-6a970cd7f11e","Type":"ContainerStarted","Data":"56abbec210fc2c91e24134ac891abb7059f60e0db2626bba5889222692a5716b"} Jul 23 17:45:18.642722 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.642699 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-n55cw" event={"ID":"f590b7bd-8791-41a3-a520-9ba3559d3371","Type":"ContainerStarted","Data":"52cbd7a54ed46b93b43b35eea498d90cc621a7fd5536fe1e2edf09307e11be21"} Jul 23 17:45:18.643961 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.643941 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-grgvm" event={"ID":"daf01600-3fe9-4f23-b700-4c66740c4978","Type":"ContainerStarted","Data":"20d00fd0d0aedeffc0d9752d996e8c6324c4d9ce97bccbc11aaa08a4e3c1038e"} Jul 23 17:45:18.645157 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.645132 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-m7k4j" event={"ID":"1cca3236-5c90-4abb-9b9b-39e40b1903c3","Type":"ContainerStarted","Data":"cd2a8dd21738805800c599ca2ee0215231e1bcb5d2f14f5e8c3be6970c69fbe9"} Jul 23 17:45:18.646400 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:18.646375 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-xttxr" event={"ID":"9b61f887-8f31-4935-9a55-626c4d3671e8","Type":"ContainerStarted","Data":"7672c42e9f4ccf1ae84b259b5e4d5243d3836ac61cc6be849826a2df453e3af7"} Jul 23 17:45:19.375247 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:19.375209 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4e92771-7bcd-405d-b9f3-c054fec7f9cf-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-v5dk6\" (UID: \"c4e92771-7bcd-405d-b9f3-c054fec7f9cf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-v5dk6" Jul 23 17:45:19.375437 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:19.375351 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8b5828c5-b41b-4b6b-8300-7879fd076ff9-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-2mbb6\" (UID: \"8b5828c5-b41b-4b6b-8300-7879fd076ff9\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2mbb6" Jul 23 17:45:19.375544 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:19.375508 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 17:45:19.375628 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:19.375570 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b5828c5-b41b-4b6b-8300-7879fd076ff9-networking-console-plugin-cert podName:8b5828c5-b41b-4b6b-8300-7879fd076ff9 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:21.37555257 +0000 UTC m=+65.453855904 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8b5828c5-b41b-4b6b-8300-7879fd076ff9-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-2mbb6" (UID: "8b5828c5-b41b-4b6b-8300-7879fd076ff9") : secret "networking-console-plugin-cert" not found Jul 23 17:45:19.375696 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:19.375651 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 17:45:19.375696 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:19.375685 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4e92771-7bcd-405d-b9f3-c054fec7f9cf-samples-operator-tls podName:c4e92771-7bcd-405d-b9f3-c054fec7f9cf nodeName:}" failed. No retries permitted until 2026-07-23 17:45:21.375674075 +0000 UTC m=+65.453977410 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c4e92771-7bcd-405d-b9f3-c054fec7f9cf-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-v5dk6" (UID: "c4e92771-7bcd-405d-b9f3-c054fec7f9cf") : secret "samples-operator-tls" not found Jul 23 17:45:19.476249 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:19.476208 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13bc8906-58cc-4237-88d6-beade4de9408-metrics-certs\") pod \"router-default-7b4c7f49cd-9kl9k\" (UID: \"13bc8906-58cc-4237-88d6-beade4de9408\") " pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:19.476412 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:19.476277 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13bc8906-58cc-4237-88d6-beade4de9408-service-ca-bundle\") pod \"router-default-7b4c7f49cd-9kl9k\" (UID: \"13bc8906-58cc-4237-88d6-beade4de9408\") " pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:19.476464 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:19.476425 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e29fa838-5b80-4246-90f9-a9095e507279-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-rwqpw\" (UID: \"e29fa838-5b80-4246-90f9-a9095e507279\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rwqpw" Jul 23 17:45:19.476619 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:19.476585 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 17:45:19.476687 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:19.476670 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e29fa838-5b80-4246-90f9-a9095e507279-cluster-monitoring-operator-tls podName:e29fa838-5b80-4246-90f9-a9095e507279 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:21.476649048 +0000 UTC m=+65.554952386 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e29fa838-5b80-4246-90f9-a9095e507279-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-rwqpw" (UID: "e29fa838-5b80-4246-90f9-a9095e507279") : secret "cluster-monitoring-operator-tls" not found Jul 23 17:45:19.477086 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:19.477064 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 17:45:19.477163 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:19.477118 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13bc8906-58cc-4237-88d6-beade4de9408-metrics-certs podName:13bc8906-58cc-4237-88d6-beade4de9408 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:21.477103717 +0000 UTC m=+65.555407051 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13bc8906-58cc-4237-88d6-beade4de9408-metrics-certs") pod "router-default-7b4c7f49cd-9kl9k" (UID: "13bc8906-58cc-4237-88d6-beade4de9408") : secret "router-metrics-certs-default" not found Jul 23 17:45:19.477225 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:19.477182 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13bc8906-58cc-4237-88d6-beade4de9408-service-ca-bundle podName:13bc8906-58cc-4237-88d6-beade4de9408 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:21.477172395 +0000 UTC m=+65.555475728 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/13bc8906-58cc-4237-88d6-beade4de9408-service-ca-bundle") pod "router-default-7b4c7f49cd-9kl9k" (UID: "13bc8906-58cc-4237-88d6-beade4de9408") : configmap references non-existent config key: service-ca.crt Jul 23 17:45:21.397218 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:21.397171 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8b5828c5-b41b-4b6b-8300-7879fd076ff9-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-2mbb6\" (UID: \"8b5828c5-b41b-4b6b-8300-7879fd076ff9\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2mbb6" Jul 23 17:45:21.397688 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:21.397304 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4e92771-7bcd-405d-b9f3-c054fec7f9cf-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-v5dk6\" (UID: \"c4e92771-7bcd-405d-b9f3-c054fec7f9cf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-v5dk6" Jul 23 17:45:21.397688 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:21.397354 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 17:45:21.397688 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:21.397431 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 17:45:21.397688 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:21.397434 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b5828c5-b41b-4b6b-8300-7879fd076ff9-networking-console-plugin-cert podName:8b5828c5-b41b-4b6b-8300-7879fd076ff9 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:25.397413453 +0000 UTC m=+69.475716790 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8b5828c5-b41b-4b6b-8300-7879fd076ff9-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-2mbb6" (UID: "8b5828c5-b41b-4b6b-8300-7879fd076ff9") : secret "networking-console-plugin-cert" not found Jul 23 17:45:21.397688 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:21.397502 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4e92771-7bcd-405d-b9f3-c054fec7f9cf-samples-operator-tls podName:c4e92771-7bcd-405d-b9f3-c054fec7f9cf nodeName:}" failed. No retries permitted until 2026-07-23 17:45:25.397484812 +0000 UTC m=+69.475788143 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c4e92771-7bcd-405d-b9f3-c054fec7f9cf-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-v5dk6" (UID: "c4e92771-7bcd-405d-b9f3-c054fec7f9cf") : secret "samples-operator-tls" not found Jul 23 17:45:21.498397 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:21.498357 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13bc8906-58cc-4237-88d6-beade4de9408-metrics-certs\") pod \"router-default-7b4c7f49cd-9kl9k\" (UID: \"13bc8906-58cc-4237-88d6-beade4de9408\") " pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:21.498647 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:21.498420 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13bc8906-58cc-4237-88d6-beade4de9408-service-ca-bundle\") pod \"router-default-7b4c7f49cd-9kl9k\" (UID: \"13bc8906-58cc-4237-88d6-beade4de9408\") " pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:21.498647 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:21.498502 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 17:45:21.498647 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:21.498554 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e29fa838-5b80-4246-90f9-a9095e507279-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-rwqpw\" (UID: \"e29fa838-5b80-4246-90f9-a9095e507279\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rwqpw" Jul 23 17:45:21.498647 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:21.498571 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13bc8906-58cc-4237-88d6-beade4de9408-service-ca-bundle podName:13bc8906-58cc-4237-88d6-beade4de9408 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:25.498550182 +0000 UTC m=+69.576853533 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/13bc8906-58cc-4237-88d6-beade4de9408-service-ca-bundle") pod "router-default-7b4c7f49cd-9kl9k" (UID: "13bc8906-58cc-4237-88d6-beade4de9408") : configmap references non-existent config key: service-ca.crt Jul 23 17:45:21.498877 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:21.498661 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 17:45:21.498877 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:21.498660 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13bc8906-58cc-4237-88d6-beade4de9408-metrics-certs podName:13bc8906-58cc-4237-88d6-beade4de9408 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:25.498618493 +0000 UTC m=+69.576921838 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13bc8906-58cc-4237-88d6-beade4de9408-metrics-certs") pod "router-default-7b4c7f49cd-9kl9k" (UID: "13bc8906-58cc-4237-88d6-beade4de9408") : secret "router-metrics-certs-default" not found Jul 23 17:45:21.498877 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:21.498725 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e29fa838-5b80-4246-90f9-a9095e507279-cluster-monitoring-operator-tls podName:e29fa838-5b80-4246-90f9-a9095e507279 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:25.498711092 +0000 UTC m=+69.577014427 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e29fa838-5b80-4246-90f9-a9095e507279-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-rwqpw" (UID: "e29fa838-5b80-4246-90f9-a9095e507279") : secret "cluster-monitoring-operator-tls" not found Jul 23 17:45:22.206476 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:22.206435 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39d55095-246a-4ecf-bb04-36a9a8c92da4-original-pull-secret\") pod \"global-pull-secret-syncer-2j6vd\" (UID: \"39d55095-246a-4ecf-bb04-36a9a8c92da4\") " pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:45:22.206693 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:22.206575 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqwqs\" (UniqueName: \"kubernetes.io/projected/cea169f8-ef99-4ce1-9559-029a6507402f-kube-api-access-dqwqs\") pod \"network-check-target-slcq6\" (UID: \"cea169f8-ef99-4ce1-9559-029a6507402f\") " pod="openshift-network-diagnostics/network-check-target-slcq6" Jul 23 17:45:22.206762 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:22.206745 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7027def-1a58-4d31-a9ce-ea9814a550a3-metrics-certs\") pod \"network-metrics-daemon-t2hlf\" (UID: \"d7027def-1a58-4d31-a9ce-ea9814a550a3\") " pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:45:22.209735 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:22.209708 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqwqs\" (UniqueName: \"kubernetes.io/projected/cea169f8-ef99-4ce1-9559-029a6507402f-kube-api-access-dqwqs\") pod \"network-check-target-slcq6\" (UID: \"cea169f8-ef99-4ce1-9559-029a6507402f\") " pod="openshift-network-diagnostics/network-check-target-slcq6" Jul 23 17:45:22.210947 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:22.210921 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 17:45:22.211106 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:22.210988 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 17:45:22.217688 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:22.217661 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 23 17:45:22.217802 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:22.217745 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7027def-1a58-4d31-a9ce-ea9814a550a3-metrics-certs podName:d7027def-1a58-4d31-a9ce-ea9814a550a3 nodeName:}" failed. No retries permitted until 2026-07-23 17:46:26.217726617 +0000 UTC m=+130.296029952 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7027def-1a58-4d31-a9ce-ea9814a550a3-metrics-certs") pod "network-metrics-daemon-t2hlf" (UID: "d7027def-1a58-4d31-a9ce-ea9814a550a3") : secret "metrics-daemon-secret" not found Jul 23 17:45:22.220345 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:22.220297 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39d55095-246a-4ecf-bb04-36a9a8c92da4-original-pull-secret\") pod \"global-pull-secret-syncer-2j6vd\" (UID: \"39d55095-246a-4ecf-bb04-36a9a8c92da4\") " pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:45:22.318381 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:22.318348 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mcdkw\"" Jul 23 17:45:22.318562 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:22.318383 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2j6vd" Jul 23 17:45:22.322850 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:22.322807 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slcq6" Jul 23 17:45:22.610067 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:22.610012 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-tls\") pod \"image-registry-dcf9785b6-pbsjf\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:45:22.610479 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:22.610173 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 17:45:22.610479 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:22.610192 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dcf9785b6-pbsjf: secret "image-registry-tls" not found Jul 23 17:45:22.610479 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:22.610258 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-tls podName:0aaf3783-02d1-4b26-a61d-b960d83d1b30 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:54.610236941 +0000 UTC m=+98.688540275 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-tls") pod "image-registry-dcf9785b6-pbsjf" (UID: "0aaf3783-02d1-4b26-a61d-b960d83d1b30") : secret "image-registry-tls" not found Jul 23 17:45:22.711363 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:22.711331 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-metrics-tls\") pod \"dns-default-vsw5b\" (UID: \"35c70c4c-f5f0-49a6-a61f-f8676deeb25a\") " pod="openshift-dns/dns-default-vsw5b" Jul 23 17:45:22.711565 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:22.711516 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 17:45:22.711664 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:22.711587 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-metrics-tls podName:35c70c4c-f5f0-49a6-a61f-f8676deeb25a nodeName:}" failed. No retries permitted until 2026-07-23 17:45:54.711566083 +0000 UTC m=+98.789869435 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-metrics-tls") pod "dns-default-vsw5b" (UID: "35c70c4c-f5f0-49a6-a61f-f8676deeb25a") : secret "dns-default-metrics-tls" not found Jul 23 17:45:23.307068 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:23.300763 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-slcq6"] Jul 23 17:45:23.329481 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:23.328415 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2j6vd"] Jul 23 17:45:23.664205 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:23.664162 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-n55cw" event={"ID":"f590b7bd-8791-41a3-a520-9ba3559d3371","Type":"ContainerStarted","Data":"116d7332ee3586f861bc866387197e069ee29d69602454523f43f49f3c2a4083"} Jul 23 17:45:23.666060 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:23.666013 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-grgvm" event={"ID":"daf01600-3fe9-4f23-b700-4c66740c4978","Type":"ContainerStarted","Data":"4f28cbfd043adcb6c8887d3b51b207abec9daa2055538c5bd3e41066e1fb05b5"} Jul 23 17:45:23.667619 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:23.667580 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-m7k4j_1cca3236-5c90-4abb-9b9b-39e40b1903c3/console-operator/0.log" Jul 23 17:45:23.667750 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:23.667639 2565 generic.go:356] "Generic (PLEG): container finished" podID="1cca3236-5c90-4abb-9b9b-39e40b1903c3" containerID="df7312ed257094ef9f7ac2b4a538f3b757d98341002262d56166b5fd026a38ab" exitCode=255 Jul 23 17:45:23.667949 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:23.667935 2565 scope.go:117] "RemoveContainer" containerID="df7312ed257094ef9f7ac2b4a538f3b757d98341002262d56166b5fd026a38ab" Jul 23 17:45:23.668194 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:23.668178 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-m7k4j" event={"ID":"1cca3236-5c90-4abb-9b9b-39e40b1903c3","Type":"ContainerDied","Data":"df7312ed257094ef9f7ac2b4a538f3b757d98341002262d56166b5fd026a38ab"} Jul 23 17:45:23.670115 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:23.670092 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-xttxr" event={"ID":"9b61f887-8f31-4935-9a55-626c4d3671e8","Type":"ContainerStarted","Data":"4874a01054b402cfd38ddaea153c98d3cd782db1f0e64733bfe8615ed5947a72"} Jul 23 17:45:23.672021 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:23.671994 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dqmhz" event={"ID":"211a908c-6578-4515-a867-8da6b0125dea","Type":"ContainerStarted","Data":"628531bfe485591902fe766ee0031e0217679aa0637011e9049a181b6966cc59"} Jul 23 17:45:23.673958 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:23.673938 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xh686" event={"ID":"e372e1b3-c8b6-4995-a770-6a970cd7f11e","Type":"ContainerStarted","Data":"f272ea3d872e264540b89847a974ab1fbfb2d7968943e1c98fed93d65afcf623"} Jul 23 17:45:23.675196 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:23.675154 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2j6vd" event={"ID":"39d55095-246a-4ecf-bb04-36a9a8c92da4","Type":"ContainerStarted","Data":"115056f71f25d391e8bc00e8d48b11b6da8d5ebfd9089c5bd866ba2d1428a991"} Jul 23 17:45:23.676916 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:23.676813 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-slcq6" event={"ID":"cea169f8-ef99-4ce1-9559-029a6507402f","Type":"ContainerStarted","Data":"e16ee1eff1eab4a9544efb3e3eaf8a9dfdc1ca0da9d5eb8832f424e70bc43938"} Jul 23 17:45:23.676916 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:23.676849 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-slcq6" event={"ID":"cea169f8-ef99-4ce1-9559-029a6507402f","Type":"ContainerStarted","Data":"78c2cdbf6edd013e939255cdf6ee4b3d273bf3123d5d23a053ec5c50c97c871c"} Jul 23 17:45:23.676916 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:23.676882 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-slcq6" Jul 23 17:45:23.683462 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:23.683387 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-n55cw" podStartSLOduration=1.876241848 podStartE2EDuration="6.68337075s" totalImagesPullingTime="4.807128902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:45:23.682373691 +0000 UTC m=+67.760677051" watchObservedRunningTime="2026-07-23 17:45:23.68337075 +0000 UTC m=+67.761674105" Jul 23 17:45:23.699510 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:23.699439 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dqmhz" podStartSLOduration=1.6624269919999999 podStartE2EDuration="6.699418632s" totalImagesPullingTime="5.03699164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:45:23.697932747 +0000 UTC m=+67.776236104" watchObservedRunningTime="2026-07-23 17:45:23.699418632 +0000 UTC m=+67.777721966" Jul 23 17:45:23.718332 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:23.718282 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-grgvm" podStartSLOduration=1.7923212670000002 podStartE2EDuration="6.718262934s" totalImagesPullingTime="4.925941667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:45:23.716204796 +0000 UTC m=+67.794508144" watchObservedRunningTime="2026-07-23 17:45:23.718262934 +0000 UTC m=+67.796566267" Jul 23 17:45:23.757046 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:23.756866 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xh686" podStartSLOduration=1.624045129 podStartE2EDuration="6.756844192s" totalImagesPullingTime="5.132799063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:45:23.736047027 +0000 UTC m=+67.814350401" watchObservedRunningTime="2026-07-23 17:45:23.756844192 +0000 UTC m=+67.835147547" Jul 23 17:45:23.757157 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:23.757053 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-xttxr" podStartSLOduration=1.8719801569999999 podStartE2EDuration="6.757045856s" totalImagesPullingTime="4.885065699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:45:23.755301658 +0000 UTC m=+67.833605012" watchObservedRunningTime="2026-07-23 17:45:23.757045856 +0000 UTC m=+67.835349211" Jul 23 17:45:23.771513 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:23.771457 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-slcq6" podStartSLOduration=67.771438023 podStartE2EDuration="1m7.771438023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:45:23.769859041 +0000 UTC m=+67.848162393" watchObservedRunningTime="2026-07-23 17:45:23.771438023 +0000 UTC m=+67.849741379" Jul 23 17:45:23.813183 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:23.813153 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kcd4h_4a324aac-61f5-46a9-b847-15730ba25434/konnectivity-agent/0.log" limit=1024 Jul 23 17:45:23.914045 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:23.914012 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kcd4h_4a324aac-61f5-46a9-b847-15730ba25434/konnectivity-agent/0.log" limit=1024 Jul 23 17:45:24.409824 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:24.408977 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9lncb"] Jul 23 17:45:24.429918 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:24.429733 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9lncb"] Jul 23 17:45:24.430925 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:24.430488 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9lncb" Jul 23 17:45:24.433568 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:24.432998 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2q846\"" Jul 23 17:45:24.433568 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:24.433231 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 23 17:45:24.433568 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:24.433418 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 23 17:45:24.452237 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:24.452194 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kcd4h_4a324aac-61f5-46a9-b847-15730ba25434/konnectivity-agent/0.log" limit=1024 Jul 23 17:45:24.518012 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:24.517911 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kcd4h_4a324aac-61f5-46a9-b847-15730ba25434/konnectivity-agent/0.log" limit=1024 Jul 23 17:45:24.534934 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:24.534902 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b73976ea-f0c0-4385-a188-01e56bac9282-crio-socket\") pod \"insights-runtime-extractor-9lncb\" (UID: \"b73976ea-f0c0-4385-a188-01e56bac9282\") " pod="openshift-insights/insights-runtime-extractor-9lncb" Jul 23 17:45:24.535149 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:24.534983 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b73976ea-f0c0-4385-a188-01e56bac9282-data-volume\") pod \"insights-runtime-extractor-9lncb\" (UID: \"b73976ea-f0c0-4385-a188-01e56bac9282\") " pod="openshift-insights/insights-runtime-extractor-9lncb" Jul 23 17:45:24.535149 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:24.535055 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b73976ea-f0c0-4385-a188-01e56bac9282-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lncb\" (UID: \"b73976ea-f0c0-4385-a188-01e56bac9282\") " pod="openshift-insights/insights-runtime-extractor-9lncb" Jul 23 17:45:24.535280 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:24.535152 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b73976ea-f0c0-4385-a188-01e56bac9282-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lncb\" (UID: \"b73976ea-f0c0-4385-a188-01e56bac9282\") " pod="openshift-insights/insights-runtime-extractor-9lncb" Jul 23 17:45:24.535280 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:24.535186 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5jzx\" (UniqueName: \"kubernetes.io/projected/b73976ea-f0c0-4385-a188-01e56bac9282-kube-api-access-h5jzx\") pod \"insights-runtime-extractor-9lncb\" (UID: \"b73976ea-f0c0-4385-a188-01e56bac9282\") " pod="openshift-insights/insights-runtime-extractor-9lncb" Jul 23 17:45:24.636184 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:24.636142 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b73976ea-f0c0-4385-a188-01e56bac9282-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lncb\" (UID: \"b73976ea-f0c0-4385-a188-01e56bac9282\") " pod="openshift-insights/insights-runtime-extractor-9lncb" Jul 23 17:45:24.636375 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:24.636203 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5jzx\" (UniqueName: \"kubernetes.io/projected/b73976ea-f0c0-4385-a188-01e56bac9282-kube-api-access-h5jzx\") pod \"insights-runtime-extractor-9lncb\" (UID: \"b73976ea-f0c0-4385-a188-01e56bac9282\") " pod="openshift-insights/insights-runtime-extractor-9lncb" Jul 23 17:45:24.636375 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:24.636280 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 17:45:24.636375 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:24.636359 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b73976ea-f0c0-4385-a188-01e56bac9282-kube-rbac-proxy-cm podName:b73976ea-f0c0-4385-a188-01e56bac9282 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:25.136335788 +0000 UTC m=+69.214639134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b73976ea-f0c0-4385-a188-01e56bac9282-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9lncb" (UID: "b73976ea-f0c0-4385-a188-01e56bac9282") : configmap "kube-rbac-proxy" not found Jul 23 17:45:24.636547 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:24.636401 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b73976ea-f0c0-4385-a188-01e56bac9282-crio-socket\") pod \"insights-runtime-extractor-9lncb\" (UID: \"b73976ea-f0c0-4385-a188-01e56bac9282\") " pod="openshift-insights/insights-runtime-extractor-9lncb" Jul 23 17:45:24.636547 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:24.636465 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b73976ea-f0c0-4385-a188-01e56bac9282-data-volume\") pod \"insights-runtime-extractor-9lncb\" (UID: \"b73976ea-f0c0-4385-a188-01e56bac9282\") " pod="openshift-insights/insights-runtime-extractor-9lncb" Jul 23 17:45:24.636547 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:24.636530 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b73976ea-f0c0-4385-a188-01e56bac9282-crio-socket\") pod \"insights-runtime-extractor-9lncb\" (UID: \"b73976ea-f0c0-4385-a188-01e56bac9282\") " pod="openshift-insights/insights-runtime-extractor-9lncb" Jul 23 17:45:24.636709 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:24.636544 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b73976ea-f0c0-4385-a188-01e56bac9282-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lncb\" (UID: \"b73976ea-f0c0-4385-a188-01e56bac9282\") " pod="openshift-insights/insights-runtime-extractor-9lncb" Jul 23 17:45:24.636709 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:24.636701 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 17:45:24.636776 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:24.636762 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b73976ea-f0c0-4385-a188-01e56bac9282-insights-runtime-extractor-tls podName:b73976ea-f0c0-4385-a188-01e56bac9282 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:25.136745284 +0000 UTC m=+69.215048621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b73976ea-f0c0-4385-a188-01e56bac9282-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9lncb" (UID: "b73976ea-f0c0-4385-a188-01e56bac9282") : secret "insights-runtime-extractor-tls" not found Jul 23 17:45:24.636860 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:24.636842 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b73976ea-f0c0-4385-a188-01e56bac9282-data-volume\") pod \"insights-runtime-extractor-9lncb\" (UID: \"b73976ea-f0c0-4385-a188-01e56bac9282\") " pod="openshift-insights/insights-runtime-extractor-9lncb" Jul 23 17:45:24.646947 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:24.646902 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5jzx\" (UniqueName: \"kubernetes.io/projected/b73976ea-f0c0-4385-a188-01e56bac9282-kube-api-access-h5jzx\") pod \"insights-runtime-extractor-9lncb\" (UID: \"b73976ea-f0c0-4385-a188-01e56bac9282\") " pod="openshift-insights/insights-runtime-extractor-9lncb" Jul 23 17:45:24.682962 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:24.682938 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-m7k4j_1cca3236-5c90-4abb-9b9b-39e40b1903c3/console-operator/1.log" Jul 23 17:45:24.683369 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:24.683350 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-m7k4j_1cca3236-5c90-4abb-9b9b-39e40b1903c3/console-operator/0.log" Jul 23 17:45:24.683438 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:24.683385 2565 generic.go:356] "Generic (PLEG): container finished" podID="1cca3236-5c90-4abb-9b9b-39e40b1903c3" containerID="8ecc904c8304c4a05487d6651223a583d0d31dac893cc7484a203bdbc81400be" exitCode=255 Jul 23 17:45:24.683513 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:24.683481 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-m7k4j" event={"ID":"1cca3236-5c90-4abb-9b9b-39e40b1903c3","Type":"ContainerDied","Data":"8ecc904c8304c4a05487d6651223a583d0d31dac893cc7484a203bdbc81400be"} Jul 23 17:45:24.683569 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:24.683546 2565 scope.go:117] "RemoveContainer" containerID="df7312ed257094ef9f7ac2b4a538f3b757d98341002262d56166b5fd026a38ab" Jul 23 17:45:24.683847 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:24.683830 2565 scope.go:117] "RemoveContainer" containerID="8ecc904c8304c4a05487d6651223a583d0d31dac893cc7484a203bdbc81400be" Jul 23 17:45:24.684368 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:24.684011 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-m7k4j_openshift-console-operator(1cca3236-5c90-4abb-9b9b-39e40b1903c3)\"" pod="openshift-console-operator/console-operator-565b7b569d-m7k4j" podUID="1cca3236-5c90-4abb-9b9b-39e40b1903c3" Jul 23 17:45:25.142058 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:25.142013 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b73976ea-f0c0-4385-a188-01e56bac9282-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lncb\" (UID: \"b73976ea-f0c0-4385-a188-01e56bac9282\") " pod="openshift-insights/insights-runtime-extractor-9lncb" Jul 23 17:45:25.142240 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:25.142127 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b73976ea-f0c0-4385-a188-01e56bac9282-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lncb\" (UID: \"b73976ea-f0c0-4385-a188-01e56bac9282\") " pod="openshift-insights/insights-runtime-extractor-9lncb" Jul 23 17:45:25.142240 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:25.142184 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 17:45:25.142350 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:25.142255 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b73976ea-f0c0-4385-a188-01e56bac9282-insights-runtime-extractor-tls podName:b73976ea-f0c0-4385-a188-01e56bac9282 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:26.14223639 +0000 UTC m=+70.220539738 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b73976ea-f0c0-4385-a188-01e56bac9282-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9lncb" (UID: "b73976ea-f0c0-4385-a188-01e56bac9282") : secret "insights-runtime-extractor-tls" not found Jul 23 17:45:25.142350 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:25.142262 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 17:45:25.142350 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:25.142337 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b73976ea-f0c0-4385-a188-01e56bac9282-kube-rbac-proxy-cm podName:b73976ea-f0c0-4385-a188-01e56bac9282 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:26.142315168 +0000 UTC m=+70.220618499 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b73976ea-f0c0-4385-a188-01e56bac9282-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9lncb" (UID: "b73976ea-f0c0-4385-a188-01e56bac9282") : configmap "kube-rbac-proxy" not found Jul 23 17:45:25.444495 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:25.444404 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8b5828c5-b41b-4b6b-8300-7879fd076ff9-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-2mbb6\" (UID: \"8b5828c5-b41b-4b6b-8300-7879fd076ff9\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2mbb6" Jul 23 17:45:25.444666 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:25.444520 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4e92771-7bcd-405d-b9f3-c054fec7f9cf-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-v5dk6\" (UID: \"c4e92771-7bcd-405d-b9f3-c054fec7f9cf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-v5dk6" Jul 23 17:45:25.444666 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:25.444616 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 17:45:25.444666 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:25.444635 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 17:45:25.444829 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:25.444692 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b5828c5-b41b-4b6b-8300-7879fd076ff9-networking-console-plugin-cert podName:8b5828c5-b41b-4b6b-8300-7879fd076ff9 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:33.444674032 +0000 UTC m=+77.522977368 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8b5828c5-b41b-4b6b-8300-7879fd076ff9-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-2mbb6" (UID: "8b5828c5-b41b-4b6b-8300-7879fd076ff9") : secret "networking-console-plugin-cert" not found Jul 23 17:45:25.444829 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:25.444712 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4e92771-7bcd-405d-b9f3-c054fec7f9cf-samples-operator-tls podName:c4e92771-7bcd-405d-b9f3-c054fec7f9cf nodeName:}" failed. No retries permitted until 2026-07-23 17:45:33.444703057 +0000 UTC m=+77.523006388 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c4e92771-7bcd-405d-b9f3-c054fec7f9cf-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-v5dk6" (UID: "c4e92771-7bcd-405d-b9f3-c054fec7f9cf") : secret "samples-operator-tls" not found Jul 23 17:45:25.502416 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:25.502380 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6nsmg_69063b0f-0422-463f-833c-5aecf8d331a7/dns-node-resolver/0.log" Jul 23 17:45:25.545319 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:25.545277 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13bc8906-58cc-4237-88d6-beade4de9408-metrics-certs\") pod \"router-default-7b4c7f49cd-9kl9k\" (UID: \"13bc8906-58cc-4237-88d6-beade4de9408\") " pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:25.545319 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:25.545320 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13bc8906-58cc-4237-88d6-beade4de9408-service-ca-bundle\") pod \"router-default-7b4c7f49cd-9kl9k\" (UID: \"13bc8906-58cc-4237-88d6-beade4de9408\") " pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:25.545566 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:25.545449 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e29fa838-5b80-4246-90f9-a9095e507279-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-rwqpw\" (UID: \"e29fa838-5b80-4246-90f9-a9095e507279\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rwqpw" Jul 23 17:45:25.545566 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:25.545469 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 17:45:25.545566 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:25.545537 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13bc8906-58cc-4237-88d6-beade4de9408-metrics-certs podName:13bc8906-58cc-4237-88d6-beade4de9408 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:33.545516404 +0000 UTC m=+77.623819737 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13bc8906-58cc-4237-88d6-beade4de9408-metrics-certs") pod "router-default-7b4c7f49cd-9kl9k" (UID: "13bc8906-58cc-4237-88d6-beade4de9408") : secret "router-metrics-certs-default" not found Jul 23 17:45:25.545566 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:25.545562 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 17:45:25.545785 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:25.545634 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e29fa838-5b80-4246-90f9-a9095e507279-cluster-monitoring-operator-tls podName:e29fa838-5b80-4246-90f9-a9095e507279 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:33.545617746 +0000 UTC m=+77.623921092 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e29fa838-5b80-4246-90f9-a9095e507279-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-rwqpw" (UID: "e29fa838-5b80-4246-90f9-a9095e507279") : secret "cluster-monitoring-operator-tls" not found Jul 23 17:45:25.545785 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:25.545660 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13bc8906-58cc-4237-88d6-beade4de9408-service-ca-bundle podName:13bc8906-58cc-4237-88d6-beade4de9408 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:33.545650454 +0000 UTC m=+77.623953788 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/13bc8906-58cc-4237-88d6-beade4de9408-service-ca-bundle") pod "router-default-7b4c7f49cd-9kl9k" (UID: "13bc8906-58cc-4237-88d6-beade4de9408") : configmap references non-existent config key: service-ca.crt Jul 23 17:45:25.603349 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:25.603316 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2wzfz"] Jul 23 17:45:25.605186 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:25.605160 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kcd4h_4a324aac-61f5-46a9-b847-15730ba25434/konnectivity-agent/0.log" limit=1024 Jul 23 17:45:25.633899 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:25.633854 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2wzfz"] Jul 23 17:45:25.634065 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:25.633998 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2wzfz" Jul 23 17:45:25.637664 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:25.637588 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-7652q\"" Jul 23 17:45:25.637664 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:25.637655 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 23 17:45:25.637924 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:25.637591 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 23 17:45:25.688797 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:25.688770 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-m7k4j_1cca3236-5c90-4abb-9b9b-39e40b1903c3/console-operator/1.log" Jul 23 17:45:25.689228 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:25.689176 2565 scope.go:117] "RemoveContainer" containerID="8ecc904c8304c4a05487d6651223a583d0d31dac893cc7484a203bdbc81400be" Jul 23 17:45:25.689382 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:25.689359 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-m7k4j_openshift-console-operator(1cca3236-5c90-4abb-9b9b-39e40b1903c3)\"" pod="openshift-console-operator/console-operator-565b7b569d-m7k4j" podUID="1cca3236-5c90-4abb-9b9b-39e40b1903c3" Jul 23 17:45:25.748786 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:25.748700 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl54m\" (UniqueName: \"kubernetes.io/projected/41a5c93d-25d0-4b6b-8978-1e9bf33daa64-kube-api-access-zl54m\") pod \"migrator-7cc9c7ffd7-2wzfz\" (UID: \"41a5c93d-25d0-4b6b-8978-1e9bf33daa64\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2wzfz" Jul 23 17:45:25.849961 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:25.849924 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl54m\" (UniqueName: \"kubernetes.io/projected/41a5c93d-25d0-4b6b-8978-1e9bf33daa64-kube-api-access-zl54m\") pod \"migrator-7cc9c7ffd7-2wzfz\" (UID: \"41a5c93d-25d0-4b6b-8978-1e9bf33daa64\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2wzfz" Jul 23 17:45:25.863721 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:25.863689 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl54m\" (UniqueName: \"kubernetes.io/projected/41a5c93d-25d0-4b6b-8978-1e9bf33daa64-kube-api-access-zl54m\") pod \"migrator-7cc9c7ffd7-2wzfz\" (UID: \"41a5c93d-25d0-4b6b-8978-1e9bf33daa64\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2wzfz" Jul 23 17:45:25.945216 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:25.945185 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2wzfz" Jul 23 17:45:26.100982 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:26.100932 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2wzfz"] Jul 23 17:45:26.103664 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:45:26.103630 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a5c93d_25d0_4b6b_8978_1e9bf33daa64.slice/crio-4d43fbfbd286aa3a85d1311da7bb24630c05829113f1ac9cd056eda144df190c WatchSource:0}: Error finding container 4d43fbfbd286aa3a85d1311da7bb24630c05829113f1ac9cd056eda144df190c: Status 404 returned error can't find the container with id 4d43fbfbd286aa3a85d1311da7bb24630c05829113f1ac9cd056eda144df190c Jul 23 17:45:26.152199 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:26.152170 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b73976ea-f0c0-4385-a188-01e56bac9282-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lncb\" (UID: \"b73976ea-f0c0-4385-a188-01e56bac9282\") " pod="openshift-insights/insights-runtime-extractor-9lncb" Jul 23 17:45:26.152370 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:26.152258 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b73976ea-f0c0-4385-a188-01e56bac9282-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lncb\" (UID: \"b73976ea-f0c0-4385-a188-01e56bac9282\") " pod="openshift-insights/insights-runtime-extractor-9lncb" Jul 23 17:45:26.152370 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:26.152336 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 17:45:26.152370 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:26.152348 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 17:45:26.152514 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:26.152415 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b73976ea-f0c0-4385-a188-01e56bac9282-kube-rbac-proxy-cm podName:b73976ea-f0c0-4385-a188-01e56bac9282 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:28.152394797 +0000 UTC m=+72.230698143 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b73976ea-f0c0-4385-a188-01e56bac9282-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9lncb" (UID: "b73976ea-f0c0-4385-a188-01e56bac9282") : configmap "kube-rbac-proxy" not found Jul 23 17:45:26.152514 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:26.152437 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b73976ea-f0c0-4385-a188-01e56bac9282-insights-runtime-extractor-tls podName:b73976ea-f0c0-4385-a188-01e56bac9282 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:28.15242743 +0000 UTC m=+72.230730774 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b73976ea-f0c0-4385-a188-01e56bac9282-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9lncb" (UID: "b73976ea-f0c0-4385-a188-01e56bac9282") : secret "insights-runtime-extractor-tls" not found Jul 23 17:45:26.503296 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:26.503272 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m8496_b11ec86f-5925-4385-aeca-5b5e88f693fa/node-ca/0.log" Jul 23 17:45:26.694517 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:26.694472 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2wzfz" event={"ID":"41a5c93d-25d0-4b6b-8978-1e9bf33daa64","Type":"ContainerStarted","Data":"4d43fbfbd286aa3a85d1311da7bb24630c05829113f1ac9cd056eda144df190c"} Jul 23 17:45:27.304833 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:27.304796 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sp8s4_5d532709-e70c-4eaf-b4e9-e650237862ea/serve-healthcheck-canary/0.log" Jul 23 17:45:28.116401 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:28.116267 2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-m7k4j" Jul 23 17:45:28.116401 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:28.116336 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-m7k4j" Jul 23 17:45:28.116888 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:28.116802 2565 scope.go:117] "RemoveContainer" containerID="8ecc904c8304c4a05487d6651223a583d0d31dac893cc7484a203bdbc81400be" Jul 23 17:45:28.117038 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:28.116996 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-m7k4j_openshift-console-operator(1cca3236-5c90-4abb-9b9b-39e40b1903c3)\"" pod="openshift-console-operator/console-operator-565b7b569d-m7k4j" podUID="1cca3236-5c90-4abb-9b9b-39e40b1903c3" Jul 23 17:45:28.139310 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:28.139238 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-wf765"] Jul 23 17:45:28.147819 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:28.147439 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-wf765" Jul 23 17:45:28.154025 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:28.153996 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 23 17:45:28.154126 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:28.154062 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 23 17:45:28.155074 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:28.155055 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-c4m7v\"" Jul 23 17:45:28.155471 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:28.155449 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 23 17:45:28.155568 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:28.155450 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 23 17:45:28.162669 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:28.162648 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-wf765"] Jul 23 17:45:28.170439 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:28.170411 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b73976ea-f0c0-4385-a188-01e56bac9282-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lncb\" (UID: \"b73976ea-f0c0-4385-a188-01e56bac9282\") " pod="openshift-insights/insights-runtime-extractor-9lncb" Jul 23 17:45:28.170544 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:28.170530 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 17:45:28.170608 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:28.170581 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b73976ea-f0c0-4385-a188-01e56bac9282-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lncb\" (UID: \"b73976ea-f0c0-4385-a188-01e56bac9282\") " pod="openshift-insights/insights-runtime-extractor-9lncb" Jul 23 17:45:28.170659 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:28.170607 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b73976ea-f0c0-4385-a188-01e56bac9282-kube-rbac-proxy-cm podName:b73976ea-f0c0-4385-a188-01e56bac9282 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:32.170577666 +0000 UTC m=+76.248880998 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b73976ea-f0c0-4385-a188-01e56bac9282-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9lncb" (UID: "b73976ea-f0c0-4385-a188-01e56bac9282") : configmap "kube-rbac-proxy" not found Jul 23 17:45:28.170726 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:28.170665 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 17:45:28.170726 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:28.170704 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b73976ea-f0c0-4385-a188-01e56bac9282-insights-runtime-extractor-tls podName:b73976ea-f0c0-4385-a188-01e56bac9282 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:32.170692183 +0000 UTC m=+76.248995513 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b73976ea-f0c0-4385-a188-01e56bac9282-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9lncb" (UID: "b73976ea-f0c0-4385-a188-01e56bac9282") : secret "insights-runtime-extractor-tls" not found Jul 23 17:45:28.272031 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:28.271984 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gjtw\" (UniqueName: \"kubernetes.io/projected/f7d4bbb2-e314-4a24-bbd6-a034ff7f939f-kube-api-access-8gjtw\") pod \"service-ca-58598695f7-wf765\" (UID: \"f7d4bbb2-e314-4a24-bbd6-a034ff7f939f\") " pod="openshift-service-ca/service-ca-58598695f7-wf765" Jul 23 17:45:28.272216 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:28.272122 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f7d4bbb2-e314-4a24-bbd6-a034ff7f939f-signing-cabundle\") pod \"service-ca-58598695f7-wf765\" (UID: \"f7d4bbb2-e314-4a24-bbd6-a034ff7f939f\") " pod="openshift-service-ca/service-ca-58598695f7-wf765" Jul 23 17:45:28.272216 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:28.272149 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f7d4bbb2-e314-4a24-bbd6-a034ff7f939f-signing-key\") pod \"service-ca-58598695f7-wf765\" (UID: \"f7d4bbb2-e314-4a24-bbd6-a034ff7f939f\") " pod="openshift-service-ca/service-ca-58598695f7-wf765" Jul 23 17:45:28.372751 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:28.372713 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gjtw\" (UniqueName: \"kubernetes.io/projected/f7d4bbb2-e314-4a24-bbd6-a034ff7f939f-kube-api-access-8gjtw\") pod \"service-ca-58598695f7-wf765\" (UID: \"f7d4bbb2-e314-4a24-bbd6-a034ff7f939f\") " pod="openshift-service-ca/service-ca-58598695f7-wf765" Jul 23 17:45:28.372908 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:28.372768 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f7d4bbb2-e314-4a24-bbd6-a034ff7f939f-signing-cabundle\") pod \"service-ca-58598695f7-wf765\" (UID: \"f7d4bbb2-e314-4a24-bbd6-a034ff7f939f\") " pod="openshift-service-ca/service-ca-58598695f7-wf765" Jul 23 17:45:28.372964 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:28.372943 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f7d4bbb2-e314-4a24-bbd6-a034ff7f939f-signing-key\") pod \"service-ca-58598695f7-wf765\" (UID: \"f7d4bbb2-e314-4a24-bbd6-a034ff7f939f\") " pod="openshift-service-ca/service-ca-58598695f7-wf765" Jul 23 17:45:28.373417 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:28.373393 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f7d4bbb2-e314-4a24-bbd6-a034ff7f939f-signing-cabundle\") pod \"service-ca-58598695f7-wf765\" (UID: \"f7d4bbb2-e314-4a24-bbd6-a034ff7f939f\") " pod="openshift-service-ca/service-ca-58598695f7-wf765" Jul 23 17:45:28.375246 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:28.375221 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f7d4bbb2-e314-4a24-bbd6-a034ff7f939f-signing-key\") pod \"service-ca-58598695f7-wf765\" (UID: \"f7d4bbb2-e314-4a24-bbd6-a034ff7f939f\") " pod="openshift-service-ca/service-ca-58598695f7-wf765" Jul 23 17:45:28.383581 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:28.383561 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gjtw\" (UniqueName: \"kubernetes.io/projected/f7d4bbb2-e314-4a24-bbd6-a034ff7f939f-kube-api-access-8gjtw\") pod \"service-ca-58598695f7-wf765\" (UID: \"f7d4bbb2-e314-4a24-bbd6-a034ff7f939f\") " pod="openshift-service-ca/service-ca-58598695f7-wf765" Jul 23 17:45:28.458620 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:28.458539 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-wf765" Jul 23 17:45:28.591491 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:28.591456 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-wf765"] Jul 23 17:45:28.703047 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:28.703007 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2j6vd" event={"ID":"39d55095-246a-4ecf-bb04-36a9a8c92da4","Type":"ContainerStarted","Data":"fe3d6cad7cb460ad1e333810dc6cd31ea62472de424789bd07797617d7c874ab"} Jul 23 17:45:28.725617 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:28.725491 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2j6vd" podStartSLOduration=67.183025738 podStartE2EDuration="1m11.725470354s" totalImagesPullingTime="4.542444616s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:44:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:45:28.723164049 +0000 UTC m=+72.801467404" watchObservedRunningTime="2026-07-23 17:45:28.725470354 +0000 UTC m=+72.803773710" Jul 23 17:45:28.773628 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:45:28.773582 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d4bbb2_e314_4a24_bbd6_a034ff7f939f.slice/crio-6ed22a60eb6c0a6b9818e1269379a0b9e2e399cd33b36efed26021c5100f8d11 WatchSource:0}: Error finding container 6ed22a60eb6c0a6b9818e1269379a0b9e2e399cd33b36efed26021c5100f8d11: Status 404 returned error can't find the container with id 6ed22a60eb6c0a6b9818e1269379a0b9e2e399cd33b36efed26021c5100f8d11 Jul 23 17:45:29.710064 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:29.710030 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2wzfz" event={"ID":"41a5c93d-25d0-4b6b-8978-1e9bf33daa64","Type":"ContainerStarted","Data":"283b7eaf6cbc9756fa638051166623543be25930dd575985997c35f55dcce056"} Jul 23 17:45:29.710529 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:29.710090 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2wzfz" event={"ID":"41a5c93d-25d0-4b6b-8978-1e9bf33daa64","Type":"ContainerStarted","Data":"2ab2cf5e37526ca96f508c9a54711dbc86d24ffcd29340921205eddcd8b654cb"} Jul 23 17:45:29.711698 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:29.711671 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-wf765" event={"ID":"f7d4bbb2-e314-4a24-bbd6-a034ff7f939f","Type":"ContainerStarted","Data":"08da26b81660bb33b60f873a3209e264b71118a9610f00b6ae5d8cf5bba56ebe"} Jul 23 17:45:29.711829 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:29.711715 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-wf765" event={"ID":"f7d4bbb2-e314-4a24-bbd6-a034ff7f939f","Type":"ContainerStarted","Data":"6ed22a60eb6c0a6b9818e1269379a0b9e2e399cd33b36efed26021c5100f8d11"} Jul 23 17:45:29.736524 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:29.736469 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2wzfz" podStartSLOduration=2.041692072 podStartE2EDuration="4.736449248s" totalImagesPullingTime="2.694757176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:45:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:45:29.733802772 +0000 UTC m=+73.812106125" watchObservedRunningTime="2026-07-23 17:45:29.736449248 +0000 UTC m=+73.814752602" Jul 23 17:45:32.211011 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:32.210976 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b73976ea-f0c0-4385-a188-01e56bac9282-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lncb\" (UID: \"b73976ea-f0c0-4385-a188-01e56bac9282\") " pod="openshift-insights/insights-runtime-extractor-9lncb" Jul 23 17:45:32.211437 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:32.211047 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b73976ea-f0c0-4385-a188-01e56bac9282-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lncb\" (UID: \"b73976ea-f0c0-4385-a188-01e56bac9282\") " pod="openshift-insights/insights-runtime-extractor-9lncb" Jul 23 17:45:32.211437 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:32.211134 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 17:45:32.211437 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:32.211148 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 17:45:32.211437 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:32.211197 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b73976ea-f0c0-4385-a188-01e56bac9282-kube-rbac-proxy-cm podName:b73976ea-f0c0-4385-a188-01e56bac9282 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:40.211181807 +0000 UTC m=+84.289485142 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b73976ea-f0c0-4385-a188-01e56bac9282-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9lncb" (UID: "b73976ea-f0c0-4385-a188-01e56bac9282") : configmap "kube-rbac-proxy" not found Jul 23 17:45:32.211437 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:32.211226 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b73976ea-f0c0-4385-a188-01e56bac9282-insights-runtime-extractor-tls podName:b73976ea-f0c0-4385-a188-01e56bac9282 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:40.211207937 +0000 UTC m=+84.289511274 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b73976ea-f0c0-4385-a188-01e56bac9282-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9lncb" (UID: "b73976ea-f0c0-4385-a188-01e56bac9282") : secret "insights-runtime-extractor-tls" not found Jul 23 17:45:33.523736 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:33.523702 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4e92771-7bcd-405d-b9f3-c054fec7f9cf-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-v5dk6\" (UID: \"c4e92771-7bcd-405d-b9f3-c054fec7f9cf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-v5dk6" Jul 23 17:45:33.524097 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:33.523816 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8b5828c5-b41b-4b6b-8300-7879fd076ff9-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-2mbb6\" (UID: \"8b5828c5-b41b-4b6b-8300-7879fd076ff9\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2mbb6" Jul 23 17:45:33.524097 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:33.523909 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 17:45:33.524097 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:33.523966 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b5828c5-b41b-4b6b-8300-7879fd076ff9-networking-console-plugin-cert podName:8b5828c5-b41b-4b6b-8300-7879fd076ff9 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:49.523951323 +0000 UTC m=+93.602254655 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8b5828c5-b41b-4b6b-8300-7879fd076ff9-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-2mbb6" (UID: "8b5828c5-b41b-4b6b-8300-7879fd076ff9") : secret "networking-console-plugin-cert" not found Jul 23 17:45:33.525941 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:33.525918 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4e92771-7bcd-405d-b9f3-c054fec7f9cf-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-v5dk6\" (UID: \"c4e92771-7bcd-405d-b9f3-c054fec7f9cf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-v5dk6" Jul 23 17:45:33.557265 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:33.557239 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-v5dk6" Jul 23 17:45:33.625218 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:33.624997 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e29fa838-5b80-4246-90f9-a9095e507279-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-rwqpw\" (UID: \"e29fa838-5b80-4246-90f9-a9095e507279\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rwqpw" Jul 23 17:45:33.625218 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:33.625078 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13bc8906-58cc-4237-88d6-beade4de9408-metrics-certs\") pod \"router-default-7b4c7f49cd-9kl9k\" (UID: \"13bc8906-58cc-4237-88d6-beade4de9408\") " pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:33.625218 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:33.625112 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13bc8906-58cc-4237-88d6-beade4de9408-service-ca-bundle\") pod \"router-default-7b4c7f49cd-9kl9k\" (UID: \"13bc8906-58cc-4237-88d6-beade4de9408\") " pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:33.625218 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:33.625150 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 17:45:33.625218 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:33.625220 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e29fa838-5b80-4246-90f9-a9095e507279-cluster-monitoring-operator-tls podName:e29fa838-5b80-4246-90f9-a9095e507279 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:49.6252052 +0000 UTC m=+93.703508532 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e29fa838-5b80-4246-90f9-a9095e507279-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-rwqpw" (UID: "e29fa838-5b80-4246-90f9-a9095e507279") : secret "cluster-monitoring-operator-tls" not found Jul 23 17:45:33.625491 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:33.625297 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13bc8906-58cc-4237-88d6-beade4de9408-service-ca-bundle podName:13bc8906-58cc-4237-88d6-beade4de9408 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:49.625285229 +0000 UTC m=+93.703588561 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/13bc8906-58cc-4237-88d6-beade4de9408-service-ca-bundle") pod "router-default-7b4c7f49cd-9kl9k" (UID: "13bc8906-58cc-4237-88d6-beade4de9408") : configmap references non-existent config key: service-ca.crt Jul 23 17:45:33.625491 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:33.625359 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 17:45:33.625491 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:33.625397 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13bc8906-58cc-4237-88d6-beade4de9408-metrics-certs podName:13bc8906-58cc-4237-88d6-beade4de9408 nodeName:}" failed. No retries permitted until 2026-07-23 17:45:49.625384137 +0000 UTC m=+93.703687470 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13bc8906-58cc-4237-88d6-beade4de9408-metrics-certs") pod "router-default-7b4c7f49cd-9kl9k" (UID: "13bc8906-58cc-4237-88d6-beade4de9408") : secret "router-metrics-certs-default" not found Jul 23 17:45:33.684380 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:33.684304 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-wf765" podStartSLOduration=5.684286241 podStartE2EDuration="5.684286241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:45:29.766894871 +0000 UTC m=+73.845198224" watchObservedRunningTime="2026-07-23 17:45:33.684286241 +0000 UTC m=+77.762589620" Jul 23 17:45:33.685115 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:33.685092 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-v5dk6"] Jul 23 17:45:34.734531 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:34.734489 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-v5dk6" event={"ID":"c4e92771-7bcd-405d-b9f3-c054fec7f9cf","Type":"ContainerStarted","Data":"7f0d41131770180f3096144b65a908037f9e304d49f9267202d96f0e643f459d"} Jul 23 17:45:35.740705 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:35.740681 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-v5dk6" event={"ID":"c4e92771-7bcd-405d-b9f3-c054fec7f9cf","Type":"ContainerStarted","Data":"9f80a8d350b955311d3b577cd9aad2199dbce857a7ec0a673f0401b9ce4de14c"} Jul 23 17:45:36.745622 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:36.745568 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-v5dk6" event={"ID":"c4e92771-7bcd-405d-b9f3-c054fec7f9cf","Type":"ContainerStarted","Data":"d51da93d097098d584d479987261912b08c24e164859e1ff56d892848f7598e8"} Jul 23 17:45:36.766877 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:36.766826 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-v5dk6" podStartSLOduration=17.826224887 podStartE2EDuration="19.766810558s" totalImagesPullingTime="1.940585671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:45:36.765994993 +0000 UTC m=+80.844298348" watchObservedRunningTime="2026-07-23 17:45:36.766810558 +0000 UTC m=+80.845113911" Jul 23 17:45:40.289877 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:40.289757 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b73976ea-f0c0-4385-a188-01e56bac9282-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lncb\" (UID: \"b73976ea-f0c0-4385-a188-01e56bac9282\") " pod="openshift-insights/insights-runtime-extractor-9lncb" Jul 23 17:45:40.290236 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:40.289976 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b73976ea-f0c0-4385-a188-01e56bac9282-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lncb\" (UID: \"b73976ea-f0c0-4385-a188-01e56bac9282\") " pod="openshift-insights/insights-runtime-extractor-9lncb" Jul 23 17:45:40.290320 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:40.290300 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b73976ea-f0c0-4385-a188-01e56bac9282-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lncb\" (UID: \"b73976ea-f0c0-4385-a188-01e56bac9282\") " pod="openshift-insights/insights-runtime-extractor-9lncb" Jul 23 17:45:40.292171 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:40.292151 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b73976ea-f0c0-4385-a188-01e56bac9282-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lncb\" (UID: \"b73976ea-f0c0-4385-a188-01e56bac9282\") " pod="openshift-insights/insights-runtime-extractor-9lncb" Jul 23 17:45:40.342761 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:40.342733 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9lncb" Jul 23 17:45:40.400066 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:40.400035 2565 scope.go:117] "RemoveContainer" containerID="8ecc904c8304c4a05487d6651223a583d0d31dac893cc7484a203bdbc81400be" Jul 23 17:45:40.480436 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:40.480411 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9lncb"] Jul 23 17:45:40.484243 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:45:40.484218 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb73976ea_f0c0_4385_a188_01e56bac9282.slice/crio-1f9755f25286ec897e850b74dac7ff2006ec918434f39deecf90b15efb537e45 WatchSource:0}: Error finding container 1f9755f25286ec897e850b74dac7ff2006ec918434f39deecf90b15efb537e45: Status 404 returned error can't find the container with id 1f9755f25286ec897e850b74dac7ff2006ec918434f39deecf90b15efb537e45 Jul 23 17:45:40.758940 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:40.758855 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9lncb" event={"ID":"b73976ea-f0c0-4385-a188-01e56bac9282","Type":"ContainerStarted","Data":"1f9755f25286ec897e850b74dac7ff2006ec918434f39deecf90b15efb537e45"} Jul 23 17:45:40.761194 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:40.761173 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-m7k4j_1cca3236-5c90-4abb-9b9b-39e40b1903c3/console-operator/1.log" Jul 23 17:45:40.761308 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:40.761240 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-m7k4j" event={"ID":"1cca3236-5c90-4abb-9b9b-39e40b1903c3","Type":"ContainerStarted","Data":"0ac052c4a430bef17f2661782997ca5ef855957e7cb827ac2385f6ee57ed8540"} Jul 23 17:45:40.761513 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:40.761491 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-m7k4j" Jul 23 17:45:40.766283 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:40.766263 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-m7k4j" Jul 23 17:45:40.782798 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:40.782753 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-m7k4j" podStartSLOduration=18.935081259 podStartE2EDuration="23.782711869s" totalImagesPullingTime="4.84763061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:45:40.781628906 +0000 UTC m=+84.859932274" watchObservedRunningTime="2026-07-23 17:45:40.782711869 +0000 UTC m=+84.861015222" Jul 23 17:45:45.780562 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:45.780530 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9lncb" event={"ID":"b73976ea-f0c0-4385-a188-01e56bac9282","Type":"ContainerStarted","Data":"8ff3035cc5b866ca358ba6f451a1cd41e9b2dfd54d1eabf48440fd8e5c0ee537"} Jul 23 17:45:49.575837 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:49.575782 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8b5828c5-b41b-4b6b-8300-7879fd076ff9-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-2mbb6\" (UID: \"8b5828c5-b41b-4b6b-8300-7879fd076ff9\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2mbb6" Jul 23 17:45:49.578266 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:49.578241 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8b5828c5-b41b-4b6b-8300-7879fd076ff9-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-2mbb6\" (UID: \"8b5828c5-b41b-4b6b-8300-7879fd076ff9\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2mbb6" Jul 23 17:45:49.676638 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:49.676569 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e29fa838-5b80-4246-90f9-a9095e507279-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-rwqpw\" (UID: \"e29fa838-5b80-4246-90f9-a9095e507279\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rwqpw" Jul 23 17:45:49.676793 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:49.676676 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13bc8906-58cc-4237-88d6-beade4de9408-metrics-certs\") pod \"router-default-7b4c7f49cd-9kl9k\" (UID: \"13bc8906-58cc-4237-88d6-beade4de9408\") " pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:49.676793 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:49.676710 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13bc8906-58cc-4237-88d6-beade4de9408-service-ca-bundle\") pod \"router-default-7b4c7f49cd-9kl9k\" (UID: \"13bc8906-58cc-4237-88d6-beade4de9408\") " pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:49.677356 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:49.677331 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13bc8906-58cc-4237-88d6-beade4de9408-service-ca-bundle\") pod \"router-default-7b4c7f49cd-9kl9k\" (UID: \"13bc8906-58cc-4237-88d6-beade4de9408\") " pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:49.678993 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:49.678971 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e29fa838-5b80-4246-90f9-a9095e507279-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-rwqpw\" (UID: \"e29fa838-5b80-4246-90f9-a9095e507279\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rwqpw" Jul 23 17:45:49.679155 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:49.679122 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13bc8906-58cc-4237-88d6-beade4de9408-metrics-certs\") pod \"router-default-7b4c7f49cd-9kl9k\" (UID: \"13bc8906-58cc-4237-88d6-beade4de9408\") " pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:49.801717 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:49.801666 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9lncb" event={"ID":"b73976ea-f0c0-4385-a188-01e56bac9282","Type":"ContainerStarted","Data":"189c2edf8685e2e9a6600cd550c41407e902ebcf306ca472e19387667bea2721"} Jul 23 17:45:49.858003 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:49.857925 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2mbb6" Jul 23 17:45:49.892978 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:49.892949 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:49.954902 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:49.954841 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rwqpw" Jul 23 17:45:50.017298 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:50.017265 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-2mbb6"] Jul 23 17:45:50.062769 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:45:50.062722 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13bc8906_58cc_4237_88d6_beade4de9408.slice/crio-8e672f7f99346832936cb6063d01b1199b890a97da8d71f384ddafa76ff5af05 WatchSource:0}: Error finding container 8e672f7f99346832936cb6063d01b1199b890a97da8d71f384ddafa76ff5af05: Status 404 returned error can't find the container with id 8e672f7f99346832936cb6063d01b1199b890a97da8d71f384ddafa76ff5af05 Jul 23 17:45:50.065193 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:50.065169 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7b4c7f49cd-9kl9k"] Jul 23 17:45:50.101884 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:50.101827 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rwqpw"] Jul 23 17:45:50.104158 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:45:50.104136 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29fa838_5b80_4246_90f9_a9095e507279.slice/crio-485020697a8352a52fe817e68e42b90d4d489cbd55df08888ff1b05e41d7d1ac WatchSource:0}: Error finding container 485020697a8352a52fe817e68e42b90d4d489cbd55df08888ff1b05e41d7d1ac: Status 404 returned error can't find the container with id 485020697a8352a52fe817e68e42b90d4d489cbd55df08888ff1b05e41d7d1ac Jul 23 17:45:50.807868 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:50.807824 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2mbb6" event={"ID":"8b5828c5-b41b-4b6b-8300-7879fd076ff9","Type":"ContainerStarted","Data":"1a23d98fd431e5775fdeba29be888b327c08248eeac683b6c7fc59156971ef52"} Jul 23 17:45:50.810122 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:50.810073 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rwqpw" event={"ID":"e29fa838-5b80-4246-90f9-a9095e507279","Type":"ContainerStarted","Data":"485020697a8352a52fe817e68e42b90d4d489cbd55df08888ff1b05e41d7d1ac"} Jul 23 17:45:50.811947 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:50.811880 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" event={"ID":"13bc8906-58cc-4237-88d6-beade4de9408","Type":"ContainerStarted","Data":"d99b71aa7cece02e355eac9d09e2e8c124127d1a9f47eeb072299b77fc453650"} Jul 23 17:45:50.811947 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:50.811926 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" event={"ID":"13bc8906-58cc-4237-88d6-beade4de9408","Type":"ContainerStarted","Data":"8e672f7f99346832936cb6063d01b1199b890a97da8d71f384ddafa76ff5af05"} Jul 23 17:45:50.846805 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:50.846753 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" podStartSLOduration=33.846737429 podStartE2EDuration="33.846737429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:45:50.846038419 +0000 UTC m=+94.924341773" watchObservedRunningTime="2026-07-23 17:45:50.846737429 +0000 UTC m=+94.925040782" Jul 23 17:45:50.893633 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:50.893584 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:50.897257 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:50.897231 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:51.468124 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.468095 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d795469c8-zcp7z"] Jul 23 17:45:51.471477 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.471451 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d795469c8-zcp7z" Jul 23 17:45:51.474834 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.474804 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7948459bf-zfwpg"] Jul 23 17:45:51.476900 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.476878 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 23 17:45:51.477202 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.477183 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 23 17:45:51.477938 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.477626 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 23 17:45:51.477938 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.477716 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 23 17:45:51.477938 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.477629 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-jpwdt\"" Jul 23 17:45:51.478273 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.478241 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f74754f98-d22mc"] Jul 23 17:45:51.478404 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.478382 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7948459bf-zfwpg" Jul 23 17:45:51.481335 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.481316 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f74754f98-d22mc" Jul 23 17:45:51.488156 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.488128 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 23 17:45:51.488623 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.488581 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 23 17:45:51.489091 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.489067 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 23 17:45:51.489267 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.489247 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 23 17:45:51.489389 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.489373 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 23 17:45:51.493085 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.493063 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b2aee9c4-3e1b-48fb-acbc-6821817336f7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6d795469c8-zcp7z\" (UID: \"b2aee9c4-3e1b-48fb-acbc-6821817336f7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d795469c8-zcp7z" Jul 23 17:45:51.493178 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.493123 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z92hd\" (UniqueName: \"kubernetes.io/projected/b2aee9c4-3e1b-48fb-acbc-6821817336f7-kube-api-access-z92hd\") pod \"managed-serviceaccount-addon-agent-6d795469c8-zcp7z\" (UID: \"b2aee9c4-3e1b-48fb-acbc-6821817336f7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d795469c8-zcp7z" Jul 23 17:45:51.505662 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.505642 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d795469c8-zcp7z"] Jul 23 17:45:51.521649 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.521581 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7948459bf-zfwpg"] Jul 23 17:45:51.523138 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.523120 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f74754f98-d22mc"] Jul 23 17:45:51.594631 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.594482 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z92hd\" (UniqueName: \"kubernetes.io/projected/b2aee9c4-3e1b-48fb-acbc-6821817336f7-kube-api-access-z92hd\") pod \"managed-serviceaccount-addon-agent-6d795469c8-zcp7z\" (UID: \"b2aee9c4-3e1b-48fb-acbc-6821817336f7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d795469c8-zcp7z" Jul 23 17:45:51.594631 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.594527 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d3d2ebc-48fd-4aff-b028-21d09efbab38-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7948459bf-zfwpg\" (UID: \"2d3d2ebc-48fd-4aff-b028-21d09efbab38\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7948459bf-zfwpg" Jul 23 17:45:51.594631 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.594562 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2d3d2ebc-48fd-4aff-b028-21d09efbab38-hub\") pod \"cluster-proxy-proxy-agent-7948459bf-zfwpg\" (UID: \"2d3d2ebc-48fd-4aff-b028-21d09efbab38\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7948459bf-zfwpg" Jul 23 17:45:51.594631 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.594589 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2d3d2ebc-48fd-4aff-b028-21d09efbab38-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7948459bf-zfwpg\" (UID: \"2d3d2ebc-48fd-4aff-b028-21d09efbab38\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7948459bf-zfwpg" Jul 23 17:45:51.594946 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.594636 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7558e342-2a04-40db-bd3d-e06016a5c27f-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f74754f98-d22mc\" (UID: \"7558e342-2a04-40db-bd3d-e06016a5c27f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f74754f98-d22mc" Jul 23 17:45:51.594946 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.594659 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2d3d2ebc-48fd-4aff-b028-21d09efbab38-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7948459bf-zfwpg\" (UID: \"2d3d2ebc-48fd-4aff-b028-21d09efbab38\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7948459bf-zfwpg" Jul 23 17:45:51.594946 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.594689 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7558e342-2a04-40db-bd3d-e06016a5c27f-tmp\") pod \"klusterlet-addon-workmgr-7f74754f98-d22mc\" (UID: \"7558e342-2a04-40db-bd3d-e06016a5c27f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f74754f98-d22mc" Jul 23 17:45:51.594946 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.594723 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2d3d2ebc-48fd-4aff-b028-21d09efbab38-ca\") pod \"cluster-proxy-proxy-agent-7948459bf-zfwpg\" (UID: \"2d3d2ebc-48fd-4aff-b028-21d09efbab38\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7948459bf-zfwpg" Jul 23 17:45:51.594946 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.594770 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddgg5\" (UniqueName: \"kubernetes.io/projected/7558e342-2a04-40db-bd3d-e06016a5c27f-kube-api-access-ddgg5\") pod \"klusterlet-addon-workmgr-7f74754f98-d22mc\" (UID: \"7558e342-2a04-40db-bd3d-e06016a5c27f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f74754f98-d22mc" Jul 23 17:45:51.594946 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.594810 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdkwg\" (UniqueName: \"kubernetes.io/projected/2d3d2ebc-48fd-4aff-b028-21d09efbab38-kube-api-access-fdkwg\") pod \"cluster-proxy-proxy-agent-7948459bf-zfwpg\" (UID: \"2d3d2ebc-48fd-4aff-b028-21d09efbab38\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7948459bf-zfwpg" Jul 23 17:45:51.595258 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.594964 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b2aee9c4-3e1b-48fb-acbc-6821817336f7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6d795469c8-zcp7z\" (UID: \"b2aee9c4-3e1b-48fb-acbc-6821817336f7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d795469c8-zcp7z" Jul 23 17:45:51.598034 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.598010 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b2aee9c4-3e1b-48fb-acbc-6821817336f7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6d795469c8-zcp7z\" (UID: \"b2aee9c4-3e1b-48fb-acbc-6821817336f7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d795469c8-zcp7z" Jul 23 17:45:51.599896 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.599871 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-jrsv7"] Jul 23 17:45:51.604699 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.604678 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-jrsv7" Jul 23 17:45:51.608653 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.608634 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 23 17:45:51.608869 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.608847 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 23 17:45:51.608965 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.608760 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-z7vqm\"" Jul 23 17:45:51.616833 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.616783 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z92hd\" (UniqueName: \"kubernetes.io/projected/b2aee9c4-3e1b-48fb-acbc-6821817336f7-kube-api-access-z92hd\") pod \"managed-serviceaccount-addon-agent-6d795469c8-zcp7z\" (UID: \"b2aee9c4-3e1b-48fb-acbc-6821817336f7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d795469c8-zcp7z" Jul 23 17:45:51.617433 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.617410 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-jrsv7"] Jul 23 17:45:51.696267 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.696234 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2d3d2ebc-48fd-4aff-b028-21d09efbab38-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7948459bf-zfwpg\" (UID: \"2d3d2ebc-48fd-4aff-b028-21d09efbab38\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7948459bf-zfwpg" Jul 23 17:45:51.696453 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.696273 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7558e342-2a04-40db-bd3d-e06016a5c27f-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f74754f98-d22mc\" (UID: \"7558e342-2a04-40db-bd3d-e06016a5c27f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f74754f98-d22mc" Jul 23 17:45:51.696453 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.696301 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2d3d2ebc-48fd-4aff-b028-21d09efbab38-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7948459bf-zfwpg\" (UID: \"2d3d2ebc-48fd-4aff-b028-21d09efbab38\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7948459bf-zfwpg" Jul 23 17:45:51.696453 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.696336 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7558e342-2a04-40db-bd3d-e06016a5c27f-tmp\") pod \"klusterlet-addon-workmgr-7f74754f98-d22mc\" (UID: \"7558e342-2a04-40db-bd3d-e06016a5c27f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f74754f98-d22mc" Jul 23 17:45:51.696453 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.696368 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2d3d2ebc-48fd-4aff-b028-21d09efbab38-ca\") pod \"cluster-proxy-proxy-agent-7948459bf-zfwpg\" (UID: \"2d3d2ebc-48fd-4aff-b028-21d09efbab38\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7948459bf-zfwpg" Jul 23 17:45:51.696453 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.696414 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddgg5\" (UniqueName: \"kubernetes.io/projected/7558e342-2a04-40db-bd3d-e06016a5c27f-kube-api-access-ddgg5\") pod \"klusterlet-addon-workmgr-7f74754f98-d22mc\" (UID: \"7558e342-2a04-40db-bd3d-e06016a5c27f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f74754f98-d22mc" Jul 23 17:45:51.696453 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.696451 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdkwg\" (UniqueName: \"kubernetes.io/projected/2d3d2ebc-48fd-4aff-b028-21d09efbab38-kube-api-access-fdkwg\") pod \"cluster-proxy-proxy-agent-7948459bf-zfwpg\" (UID: \"2d3d2ebc-48fd-4aff-b028-21d09efbab38\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7948459bf-zfwpg" Jul 23 17:45:51.696811 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.696526 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d3d2ebc-48fd-4aff-b028-21d09efbab38-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7948459bf-zfwpg\" (UID: \"2d3d2ebc-48fd-4aff-b028-21d09efbab38\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7948459bf-zfwpg" Jul 23 17:45:51.696811 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.696564 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2d3d2ebc-48fd-4aff-b028-21d09efbab38-hub\") pod \"cluster-proxy-proxy-agent-7948459bf-zfwpg\" (UID: \"2d3d2ebc-48fd-4aff-b028-21d09efbab38\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7948459bf-zfwpg" Jul 23 17:45:51.696811 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.696590 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlgr8\" (UniqueName: \"kubernetes.io/projected/6a7d3768-fb48-410a-b5cb-e936bdfb9baf-kube-api-access-nlgr8\") pod \"downloads-5bc67dcd46-jrsv7\" (UID: \"6a7d3768-fb48-410a-b5cb-e936bdfb9baf\") " pod="openshift-console/downloads-5bc67dcd46-jrsv7" Jul 23 17:45:51.696966 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.696923 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7558e342-2a04-40db-bd3d-e06016a5c27f-tmp\") pod \"klusterlet-addon-workmgr-7f74754f98-d22mc\" (UID: \"7558e342-2a04-40db-bd3d-e06016a5c27f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f74754f98-d22mc" Jul 23 17:45:51.697861 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.697832 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d3d2ebc-48fd-4aff-b028-21d09efbab38-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7948459bf-zfwpg\" (UID: \"2d3d2ebc-48fd-4aff-b028-21d09efbab38\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7948459bf-zfwpg" Jul 23 17:45:51.699479 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.699448 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2d3d2ebc-48fd-4aff-b028-21d09efbab38-ca\") pod \"cluster-proxy-proxy-agent-7948459bf-zfwpg\" (UID: \"2d3d2ebc-48fd-4aff-b028-21d09efbab38\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7948459bf-zfwpg" Jul 23 17:45:51.699620 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.699569 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2d3d2ebc-48fd-4aff-b028-21d09efbab38-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7948459bf-zfwpg\" (UID: \"2d3d2ebc-48fd-4aff-b028-21d09efbab38\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7948459bf-zfwpg" Jul 23 17:45:51.699812 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.699794 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2d3d2ebc-48fd-4aff-b028-21d09efbab38-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7948459bf-zfwpg\" (UID: \"2d3d2ebc-48fd-4aff-b028-21d09efbab38\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7948459bf-zfwpg" Jul 23 17:45:51.699892 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.699874 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2d3d2ebc-48fd-4aff-b028-21d09efbab38-hub\") pod \"cluster-proxy-proxy-agent-7948459bf-zfwpg\" (UID: \"2d3d2ebc-48fd-4aff-b028-21d09efbab38\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7948459bf-zfwpg" Jul 23 17:45:51.699942 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.699892 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7558e342-2a04-40db-bd3d-e06016a5c27f-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f74754f98-d22mc\" (UID: \"7558e342-2a04-40db-bd3d-e06016a5c27f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f74754f98-d22mc" Jul 23 17:45:51.710379 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.710359 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddgg5\" (UniqueName: \"kubernetes.io/projected/7558e342-2a04-40db-bd3d-e06016a5c27f-kube-api-access-ddgg5\") pod \"klusterlet-addon-workmgr-7f74754f98-d22mc\" (UID: \"7558e342-2a04-40db-bd3d-e06016a5c27f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f74754f98-d22mc" Jul 23 17:45:51.710525 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.710496 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdkwg\" (UniqueName: \"kubernetes.io/projected/2d3d2ebc-48fd-4aff-b028-21d09efbab38-kube-api-access-fdkwg\") pod \"cluster-proxy-proxy-agent-7948459bf-zfwpg\" (UID: \"2d3d2ebc-48fd-4aff-b028-21d09efbab38\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7948459bf-zfwpg" Jul 23 17:45:51.790267 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.790231 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d795469c8-zcp7z" Jul 23 17:45:51.797761 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.797735 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7948459bf-zfwpg" Jul 23 17:45:51.798096 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.798067 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlgr8\" (UniqueName: \"kubernetes.io/projected/6a7d3768-fb48-410a-b5cb-e936bdfb9baf-kube-api-access-nlgr8\") pod \"downloads-5bc67dcd46-jrsv7\" (UID: \"6a7d3768-fb48-410a-b5cb-e936bdfb9baf\") " pod="openshift-console/downloads-5bc67dcd46-jrsv7" Jul 23 17:45:51.813421 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.813397 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlgr8\" (UniqueName: \"kubernetes.io/projected/6a7d3768-fb48-410a-b5cb-e936bdfb9baf-kube-api-access-nlgr8\") pod \"downloads-5bc67dcd46-jrsv7\" (UID: \"6a7d3768-fb48-410a-b5cb-e936bdfb9baf\") " pod="openshift-console/downloads-5bc67dcd46-jrsv7" Jul 23 17:45:51.817243 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.817204 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2mbb6" event={"ID":"8b5828c5-b41b-4b6b-8300-7879fd076ff9","Type":"ContainerStarted","Data":"bc407bc0dcbf277da695b33ef58490092bb1fdaa241b21440f4ddd6fa23a092f"} Jul 23 17:45:51.817374 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.817333 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:51.818858 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.818814 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7b4c7f49cd-9kl9k" Jul 23 17:45:51.820812 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.820793 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f74754f98-d22mc" Jul 23 17:45:51.837756 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.837714 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-2mbb6" podStartSLOduration=33.578538719 podStartE2EDuration="34.837699115s" totalImagesPullingTime="1.259160396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:45:51.837040581 +0000 UTC m=+95.915343969" watchObservedRunningTime="2026-07-23 17:45:51.837699115 +0000 UTC m=+95.916002468" Jul 23 17:45:51.916874 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:51.916799 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-jrsv7" Jul 23 17:45:52.237637 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:52.237579 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d795469c8-zcp7z"] Jul 23 17:45:52.241885 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:45:52.241855 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2aee9c4_3e1b_48fb_acbc_6821817336f7.slice/crio-de68ff22f719251139adbc2f768e58c06322a3714f5eba734fe0dfaecc2e7d56 WatchSource:0}: Error finding container de68ff22f719251139adbc2f768e58c06322a3714f5eba734fe0dfaecc2e7d56: Status 404 returned error can't find the container with id de68ff22f719251139adbc2f768e58c06322a3714f5eba734fe0dfaecc2e7d56 Jul 23 17:45:52.256107 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:52.256026 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-jrsv7"] Jul 23 17:45:52.259707 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:45:52.259679 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7d3768_fb48_410a_b5cb_e936bdfb9baf.slice/crio-e4ef62c0931bf57e7d26c22d68b0387af29ac955fc3564426515b13fb0ea020e WatchSource:0}: Error finding container e4ef62c0931bf57e7d26c22d68b0387af29ac955fc3564426515b13fb0ea020e: Status 404 returned error can't find the container with id e4ef62c0931bf57e7d26c22d68b0387af29ac955fc3564426515b13fb0ea020e Jul 23 17:45:52.501872 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:52.501830 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7948459bf-zfwpg"] Jul 23 17:45:52.506487 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:52.506457 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f74754f98-d22mc"] Jul 23 17:45:52.510440 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:45:52.510402 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7558e342_2a04_40db_bd3d_e06016a5c27f.slice/crio-8e50aca3069add8280d050d928d13c72c0e943327b092878ce4aa0b6510173ee WatchSource:0}: Error finding container 8e50aca3069add8280d050d928d13c72c0e943327b092878ce4aa0b6510173ee: Status 404 returned error can't find the container with id 8e50aca3069add8280d050d928d13c72c0e943327b092878ce4aa0b6510173ee Jul 23 17:45:52.693366 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:52.693333 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-2tmcd"] Jul 23 17:45:52.697062 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:52.697036 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-2tmcd" Jul 23 17:45:52.700094 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:52.700070 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 23 17:45:52.700231 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:52.700098 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-p8twx\"" Jul 23 17:45:52.706638 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:52.706557 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-2tmcd"] Jul 23 17:45:52.809728 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:52.809691 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f608c6e6-9cbe-45f5-b2d6-b1f19b846bbd-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-2tmcd\" (UID: \"f608c6e6-9cbe-45f5-b2d6-b1f19b846bbd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-2tmcd" Jul 23 17:45:52.824118 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:52.824081 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rwqpw" event={"ID":"e29fa838-5b80-4246-90f9-a9095e507279","Type":"ContainerStarted","Data":"381225d80a4d0c4ac70d421fa3805d1a292f64892453d287346ed1fd4d1625af"} Jul 23 17:45:52.826326 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:52.826276 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f74754f98-d22mc" event={"ID":"7558e342-2a04-40db-bd3d-e06016a5c27f","Type":"ContainerStarted","Data":"8e50aca3069add8280d050d928d13c72c0e943327b092878ce4aa0b6510173ee"} Jul 23 17:45:52.828007 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:52.827750 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7948459bf-zfwpg" event={"ID":"2d3d2ebc-48fd-4aff-b028-21d09efbab38","Type":"ContainerStarted","Data":"39b8c86852bd30dc588d5f2f64eb492bf9d7b826013808d13dd787aaab8f76b7"} Jul 23 17:45:52.829137 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:52.829112 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d795469c8-zcp7z" event={"ID":"b2aee9c4-3e1b-48fb-acbc-6821817336f7","Type":"ContainerStarted","Data":"de68ff22f719251139adbc2f768e58c06322a3714f5eba734fe0dfaecc2e7d56"} Jul 23 17:45:52.831330 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:52.831303 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-jrsv7" event={"ID":"6a7d3768-fb48-410a-b5cb-e936bdfb9baf","Type":"ContainerStarted","Data":"e4ef62c0931bf57e7d26c22d68b0387af29ac955fc3564426515b13fb0ea020e"} Jul 23 17:45:52.844623 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:52.844046 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-rwqpw" podStartSLOduration=33.822074404 podStartE2EDuration="35.844028186s" totalImagesPullingTime="2.021953782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:45:52.841954144 +0000 UTC m=+96.920257521" watchObservedRunningTime="2026-07-23 17:45:52.844028186 +0000 UTC m=+96.922331541" Jul 23 17:45:52.912774 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:52.910953 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f608c6e6-9cbe-45f5-b2d6-b1f19b846bbd-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-2tmcd\" (UID: \"f608c6e6-9cbe-45f5-b2d6-b1f19b846bbd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-2tmcd" Jul 23 17:45:52.912774 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:52.912432 2565 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 23 17:45:52.912774 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:45:52.912493 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f608c6e6-9cbe-45f5-b2d6-b1f19b846bbd-tls-certificates podName:f608c6e6-9cbe-45f5-b2d6-b1f19b846bbd nodeName:}" failed. No retries permitted until 2026-07-23 17:45:53.412475367 +0000 UTC m=+97.490778703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/f608c6e6-9cbe-45f5-b2d6-b1f19b846bbd-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-2tmcd" (UID: "f608c6e6-9cbe-45f5-b2d6-b1f19b846bbd") : secret "prometheus-operator-admission-webhook-tls" not found Jul 23 17:45:53.417066 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:53.416974 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f608c6e6-9cbe-45f5-b2d6-b1f19b846bbd-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-2tmcd\" (UID: \"f608c6e6-9cbe-45f5-b2d6-b1f19b846bbd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-2tmcd" Jul 23 17:45:53.423890 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:53.423834 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f608c6e6-9cbe-45f5-b2d6-b1f19b846bbd-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-2tmcd\" (UID: \"f608c6e6-9cbe-45f5-b2d6-b1f19b846bbd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-2tmcd" Jul 23 17:45:53.611610 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:53.611299 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-2tmcd" Jul 23 17:45:53.885546 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:53.885458 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-2tmcd"] Jul 23 17:45:54.628616 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:54.628131 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-tls\") pod \"image-registry-dcf9785b6-pbsjf\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:45:54.639915 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:54.639881 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-tls\") pod \"image-registry-dcf9785b6-pbsjf\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:45:54.687641 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:54.686815 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-slcq6" Jul 23 17:45:54.699366 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:54.699334 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s7c84\"" Jul 23 17:45:54.707902 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:54.707665 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:45:54.731631 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:54.729134 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-metrics-tls\") pod \"dns-default-vsw5b\" (UID: \"35c70c4c-f5f0-49a6-a61f-f8676deeb25a\") " pod="openshift-dns/dns-default-vsw5b" Jul 23 17:45:54.738370 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:54.736680 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35c70c4c-f5f0-49a6-a61f-f8676deeb25a-metrics-tls\") pod \"dns-default-vsw5b\" (UID: \"35c70c4c-f5f0-49a6-a61f-f8676deeb25a\") " pod="openshift-dns/dns-default-vsw5b" Jul 23 17:45:54.853470 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:54.853347 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-2tmcd" event={"ID":"f608c6e6-9cbe-45f5-b2d6-b1f19b846bbd","Type":"ContainerStarted","Data":"a8811eff21917aaf8f77e99b136174dab3d9a581305a6aa5446c7c89c9146b9b"} Jul 23 17:45:54.968611 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:54.968268 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-dcf9785b6-pbsjf"] Jul 23 17:45:55.008930 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:55.008735 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bjl7v\"" Jul 23 17:45:55.016983 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:55.016740 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vsw5b" Jul 23 17:45:55.862590 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:55.862240 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" event={"ID":"0aaf3783-02d1-4b26-a61d-b960d83d1b30","Type":"ContainerStarted","Data":"f2527a09269fded234580c736f663fa8c4fa47002594faaeb9c8554a9d295213"} Jul 23 17:45:55.862590 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:55.862301 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" event={"ID":"0aaf3783-02d1-4b26-a61d-b960d83d1b30","Type":"ContainerStarted","Data":"5966a4d67c4e805a1910902a5d9349a21fa9c1d6555a7030776ed82ad8cee346"} Jul 23 17:45:55.862590 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:55.862545 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:45:57.064529 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:57.064475 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" podStartSLOduration=101.064455717 podStartE2EDuration="1m41.064455717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:45:55.885882923 +0000 UTC m=+99.964186276" watchObservedRunningTime="2026-07-23 17:45:57.064455717 +0000 UTC m=+101.142759071" Jul 23 17:45:57.065166 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:57.065145 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-ff97bc65f-5fvcc"] Jul 23 17:45:57.089348 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:57.089321 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ff97bc65f-5fvcc"] Jul 23 17:45:57.089509 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:57.089447 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff97bc65f-5fvcc" Jul 23 17:45:57.092063 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:57.091996 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 23 17:45:57.092225 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:57.092096 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 23 17:45:57.092440 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:57.092416 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 23 17:45:57.092577 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:57.092557 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 23 17:45:57.093051 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:57.093032 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 23 17:45:57.093183 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:57.093146 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-78r8z\"" Jul 23 17:45:57.157258 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:57.157221 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-console-oauth-config\") pod \"console-ff97bc65f-5fvcc\" (UID: \"6a9979ef-2936-4c14-8ef1-2b0393f5abcc\") " pod="openshift-console/console-ff97bc65f-5fvcc" Jul 23 17:45:57.157441 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:57.157273 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-console-serving-cert\") pod \"console-ff97bc65f-5fvcc\" (UID: \"6a9979ef-2936-4c14-8ef1-2b0393f5abcc\") " pod="openshift-console/console-ff97bc65f-5fvcc" Jul 23 17:45:57.157441 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:57.157303 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-console-config\") pod \"console-ff97bc65f-5fvcc\" (UID: \"6a9979ef-2936-4c14-8ef1-2b0393f5abcc\") " pod="openshift-console/console-ff97bc65f-5fvcc" Jul 23 17:45:57.157441 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:57.157334 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fj4n\" (UniqueName: \"kubernetes.io/projected/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-kube-api-access-8fj4n\") pod \"console-ff97bc65f-5fvcc\" (UID: \"6a9979ef-2936-4c14-8ef1-2b0393f5abcc\") " pod="openshift-console/console-ff97bc65f-5fvcc" Jul 23 17:45:57.157441 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:57.157363 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-service-ca\") pod \"console-ff97bc65f-5fvcc\" (UID: \"6a9979ef-2936-4c14-8ef1-2b0393f5abcc\") " pod="openshift-console/console-ff97bc65f-5fvcc" Jul 23 17:45:57.157675 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:57.157471 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-oauth-serving-cert\") pod \"console-ff97bc65f-5fvcc\" (UID: \"6a9979ef-2936-4c14-8ef1-2b0393f5abcc\") " pod="openshift-console/console-ff97bc65f-5fvcc" Jul 23 17:45:57.258479 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:57.258442 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-console-serving-cert\") pod \"console-ff97bc65f-5fvcc\" (UID: \"6a9979ef-2936-4c14-8ef1-2b0393f5abcc\") " pod="openshift-console/console-ff97bc65f-5fvcc" Jul 23 17:45:57.258479 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:57.258489 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-console-config\") pod \"console-ff97bc65f-5fvcc\" (UID: \"6a9979ef-2936-4c14-8ef1-2b0393f5abcc\") " pod="openshift-console/console-ff97bc65f-5fvcc" Jul 23 17:45:57.258740 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:57.258527 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fj4n\" (UniqueName: \"kubernetes.io/projected/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-kube-api-access-8fj4n\") pod \"console-ff97bc65f-5fvcc\" (UID: \"6a9979ef-2936-4c14-8ef1-2b0393f5abcc\") " pod="openshift-console/console-ff97bc65f-5fvcc" Jul 23 17:45:57.258740 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:57.258564 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-service-ca\") pod \"console-ff97bc65f-5fvcc\" (UID: \"6a9979ef-2936-4c14-8ef1-2b0393f5abcc\") " pod="openshift-console/console-ff97bc65f-5fvcc" Jul 23 17:45:57.258740 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:57.258639 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-oauth-serving-cert\") pod \"console-ff97bc65f-5fvcc\" (UID: \"6a9979ef-2936-4c14-8ef1-2b0393f5abcc\") " pod="openshift-console/console-ff97bc65f-5fvcc" Jul 23 17:45:57.258740 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:57.258730 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-console-oauth-config\") pod \"console-ff97bc65f-5fvcc\" (UID: \"6a9979ef-2936-4c14-8ef1-2b0393f5abcc\") " pod="openshift-console/console-ff97bc65f-5fvcc" Jul 23 17:45:57.259529 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:57.259499 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-console-config\") pod \"console-ff97bc65f-5fvcc\" (UID: \"6a9979ef-2936-4c14-8ef1-2b0393f5abcc\") " pod="openshift-console/console-ff97bc65f-5fvcc" Jul 23 17:45:57.259675 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:57.259501 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-service-ca\") pod \"console-ff97bc65f-5fvcc\" (UID: \"6a9979ef-2936-4c14-8ef1-2b0393f5abcc\") " pod="openshift-console/console-ff97bc65f-5fvcc" Jul 23 17:45:57.259675 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:57.259626 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-oauth-serving-cert\") pod \"console-ff97bc65f-5fvcc\" (UID: \"6a9979ef-2936-4c14-8ef1-2b0393f5abcc\") " pod="openshift-console/console-ff97bc65f-5fvcc" Jul 23 17:45:57.261885 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:57.261865 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-console-serving-cert\") pod \"console-ff97bc65f-5fvcc\" (UID: \"6a9979ef-2936-4c14-8ef1-2b0393f5abcc\") " pod="openshift-console/console-ff97bc65f-5fvcc" Jul 23 17:45:57.261989 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:57.261884 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-console-oauth-config\") pod \"console-ff97bc65f-5fvcc\" (UID: \"6a9979ef-2936-4c14-8ef1-2b0393f5abcc\") " pod="openshift-console/console-ff97bc65f-5fvcc" Jul 23 17:45:57.269120 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:57.269088 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fj4n\" (UniqueName: \"kubernetes.io/projected/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-kube-api-access-8fj4n\") pod \"console-ff97bc65f-5fvcc\" (UID: \"6a9979ef-2936-4c14-8ef1-2b0393f5abcc\") " pod="openshift-console/console-ff97bc65f-5fvcc" Jul 23 17:45:57.400403 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:45:57.400315 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff97bc65f-5fvcc" Jul 23 17:46:02.742290 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:02.742253 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vsw5b"] Jul 23 17:46:02.768343 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:02.768310 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ff97bc65f-5fvcc"] Jul 23 17:46:02.917936 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:46:02.917857 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35c70c4c_f5f0_49a6_a61f_f8676deeb25a.slice/crio-41a3582f7e06db71e65d847fec18e2b2dc769493305b565cc8c79ded4d500028 WatchSource:0}: Error finding container 41a3582f7e06db71e65d847fec18e2b2dc769493305b565cc8c79ded4d500028: Status 404 returned error can't find the container with id 41a3582f7e06db71e65d847fec18e2b2dc769493305b565cc8c79ded4d500028 Jul 23 17:46:02.918834 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:46:02.918808 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a9979ef_2936_4c14_8ef1_2b0393f5abcc.slice/crio-5b157663c4bf11c56cba7d173a5c2562b62d48806cf523d9438b1a99619987a3 WatchSource:0}: Error finding container 5b157663c4bf11c56cba7d173a5c2562b62d48806cf523d9438b1a99619987a3: Status 404 returned error can't find the container with id 5b157663c4bf11c56cba7d173a5c2562b62d48806cf523d9438b1a99619987a3 Jul 23 17:46:03.904841 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:03.904789 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9lncb" event={"ID":"b73976ea-f0c0-4385-a188-01e56bac9282","Type":"ContainerStarted","Data":"88c336a99fdd209d7de6b7aad4ba543810eb1d426c3174953683520999446c33"} Jul 23 17:46:03.909393 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:03.909296 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7948459bf-zfwpg" event={"ID":"2d3d2ebc-48fd-4aff-b028-21d09efbab38","Type":"ContainerStarted","Data":"48672194cce779ad85185e741f93bef9837a5295bf25e66f947a77e70d7df22b"} Jul 23 17:46:03.909393 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:03.909344 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7948459bf-zfwpg" event={"ID":"2d3d2ebc-48fd-4aff-b028-21d09efbab38","Type":"ContainerStarted","Data":"ba69395603b53fae00583a349a095ef19beb795c9e155a17180b8e90a1a447bd"} Jul 23 17:46:03.909393 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:03.909363 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7948459bf-zfwpg" event={"ID":"2d3d2ebc-48fd-4aff-b028-21d09efbab38","Type":"ContainerStarted","Data":"e88e0b26d0d2b6e550396073ee00f728dc86f5df98102d655e60df2a1898a7f2"} Jul 23 17:46:03.910930 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:03.910901 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d795469c8-zcp7z" event={"ID":"b2aee9c4-3e1b-48fb-acbc-6821817336f7","Type":"ContainerStarted","Data":"0a8105a24c4ed62af1dc06bb2653e55610ec376c31dc1002bf8293c02ce141a4"} Jul 23 17:46:03.912679 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:03.912652 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff97bc65f-5fvcc" event={"ID":"6a9979ef-2936-4c14-8ef1-2b0393f5abcc","Type":"ContainerStarted","Data":"5b157663c4bf11c56cba7d173a5c2562b62d48806cf523d9438b1a99619987a3"} Jul 23 17:46:03.914362 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:03.914338 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-2tmcd" event={"ID":"f608c6e6-9cbe-45f5-b2d6-b1f19b846bbd","Type":"ContainerStarted","Data":"83da97644ebaa665b6fc5b7bb911a52d9507bcaa6ccbb66ec4d2cd81a04e1f4b"} Jul 23 17:46:03.914520 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:03.914499 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-2tmcd" Jul 23 17:46:03.915967 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:03.915901 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vsw5b" event={"ID":"35c70c4c-f5f0-49a6-a61f-f8676deeb25a","Type":"ContainerStarted","Data":"41a3582f7e06db71e65d847fec18e2b2dc769493305b565cc8c79ded4d500028"} Jul 23 17:46:03.918266 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:03.918223 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f74754f98-d22mc" event={"ID":"7558e342-2a04-40db-bd3d-e06016a5c27f","Type":"ContainerStarted","Data":"6d7e7ffce6623b74fd5f2848a78202c323eef1b7cd6a517842a6dc5c60c7dc6b"} Jul 23 17:46:03.918718 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:03.918674 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f74754f98-d22mc" Jul 23 17:46:03.920580 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:03.920531 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f74754f98-d22mc" Jul 23 17:46:03.920688 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:03.920650 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-2tmcd" Jul 23 17:46:03.936490 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:03.936423 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9lncb" podStartSLOduration=17.609215743 podStartE2EDuration="39.936403801s" totalImagesPullingTime="22.327188058s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:45:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:46:03.935406074 +0000 UTC m=+108.013709452" watchObservedRunningTime="2026-07-23 17:46:03.936403801 +0000 UTC m=+108.014707156" Jul 23 17:46:03.963321 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:03.963221 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d795469c8-zcp7z" podStartSLOduration=2.632394483 podStartE2EDuration="12.963201852s" totalImagesPullingTime="10.330807369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:45:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:46:03.962365301 +0000 UTC m=+108.040668654" watchObservedRunningTime="2026-07-23 17:46:03.963201852 +0000 UTC m=+108.041505201" Jul 23 17:46:03.989504 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:03.989386 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7948459bf-zfwpg" podStartSLOduration=2.580685137 podStartE2EDuration="12.989369833s" totalImagesPullingTime="10.408684696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:45:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:46:03.987544369 +0000 UTC m=+108.065847746" watchObservedRunningTime="2026-07-23 17:46:03.989369833 +0000 UTC m=+108.067673186" Jul 23 17:46:04.029152 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:04.028931 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f74754f98-d22mc" podStartSLOduration=2.622827402 podStartE2EDuration="13.028910384s" totalImagesPullingTime="10.406082982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:45:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:46:04.008911992 +0000 UTC m=+108.087215346" watchObservedRunningTime="2026-07-23 17:46:04.028910384 +0000 UTC m=+108.107213739" Jul 23 17:46:04.030067 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:04.029257 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-2tmcd" podStartSLOduration=2.96272279 podStartE2EDuration="12.029248315s" totalImagesPullingTime="9.066525525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:45:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:46:04.02825439 +0000 UTC m=+108.106557782" watchObservedRunningTime="2026-07-23 17:46:04.029248315 +0000 UTC m=+108.107551650" Jul 23 17:46:05.931390 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:05.931354 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vsw5b" event={"ID":"35c70c4c-f5f0-49a6-a61f-f8676deeb25a","Type":"ContainerStarted","Data":"b6719da6da11ed99c377772639fc252ab5a5d458497040f3f2471ae3e3565f2c"} Jul 23 17:46:12.206490 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.206454 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hcvz7"] Jul 23 17:46:12.297010 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.296975 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hcvz7" Jul 23 17:46:12.310385 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.310359 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 23 17:46:12.310531 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.310388 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 23 17:46:12.310621 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.310546 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 23 17:46:12.310721 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.310702 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 23 17:46:12.314913 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.314894 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xxs4p\"" Jul 23 17:46:12.403753 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.403715 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1eb42749-defc-4272-a671-4569a9705519-metrics-client-ca\") pod \"node-exporter-hcvz7\" (UID: \"1eb42749-defc-4272-a671-4569a9705519\") " pod="openshift-monitoring/node-exporter-hcvz7" Jul 23 17:46:12.403918 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.403798 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1eb42749-defc-4272-a671-4569a9705519-sys\") pod \"node-exporter-hcvz7\" (UID: \"1eb42749-defc-4272-a671-4569a9705519\") " pod="openshift-monitoring/node-exporter-hcvz7" Jul 23 17:46:12.403918 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.403823 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnr2w\" (UniqueName: \"kubernetes.io/projected/1eb42749-defc-4272-a671-4569a9705519-kube-api-access-qnr2w\") pod \"node-exporter-hcvz7\" (UID: \"1eb42749-defc-4272-a671-4569a9705519\") " pod="openshift-monitoring/node-exporter-hcvz7" Jul 23 17:46:12.403918 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.403849 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1eb42749-defc-4272-a671-4569a9705519-node-exporter-textfile\") pod \"node-exporter-hcvz7\" (UID: \"1eb42749-defc-4272-a671-4569a9705519\") " pod="openshift-monitoring/node-exporter-hcvz7" Jul 23 17:46:12.404085 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.403938 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1eb42749-defc-4272-a671-4569a9705519-root\") pod \"node-exporter-hcvz7\" (UID: \"1eb42749-defc-4272-a671-4569a9705519\") " pod="openshift-monitoring/node-exporter-hcvz7" Jul 23 17:46:12.404085 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.403990 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1eb42749-defc-4272-a671-4569a9705519-node-exporter-tls\") pod \"node-exporter-hcvz7\" (UID: \"1eb42749-defc-4272-a671-4569a9705519\") " pod="openshift-monitoring/node-exporter-hcvz7" Jul 23 17:46:12.404085 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.404023 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1eb42749-defc-4272-a671-4569a9705519-node-exporter-wtmp\") pod \"node-exporter-hcvz7\" (UID: \"1eb42749-defc-4272-a671-4569a9705519\") " pod="openshift-monitoring/node-exporter-hcvz7" Jul 23 17:46:12.404085 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.404060 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1eb42749-defc-4272-a671-4569a9705519-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hcvz7\" (UID: \"1eb42749-defc-4272-a671-4569a9705519\") " pod="openshift-monitoring/node-exporter-hcvz7" Jul 23 17:46:12.404227 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.404097 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1eb42749-defc-4272-a671-4569a9705519-node-exporter-accelerators-collector-config\") pod \"node-exporter-hcvz7\" (UID: \"1eb42749-defc-4272-a671-4569a9705519\") " pod="openshift-monitoring/node-exporter-hcvz7" Jul 23 17:46:12.505235 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.505155 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1eb42749-defc-4272-a671-4569a9705519-metrics-client-ca\") pod \"node-exporter-hcvz7\" (UID: \"1eb42749-defc-4272-a671-4569a9705519\") " pod="openshift-monitoring/node-exporter-hcvz7" Jul 23 17:46:12.505235 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.505220 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1eb42749-defc-4272-a671-4569a9705519-sys\") pod \"node-exporter-hcvz7\" (UID: \"1eb42749-defc-4272-a671-4569a9705519\") " pod="openshift-monitoring/node-exporter-hcvz7" Jul 23 17:46:12.505467 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.505255 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnr2w\" (UniqueName: \"kubernetes.io/projected/1eb42749-defc-4272-a671-4569a9705519-kube-api-access-qnr2w\") pod \"node-exporter-hcvz7\" (UID: \"1eb42749-defc-4272-a671-4569a9705519\") " pod="openshift-monitoring/node-exporter-hcvz7" Jul 23 17:46:12.505467 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.505292 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1eb42749-defc-4272-a671-4569a9705519-node-exporter-textfile\") pod \"node-exporter-hcvz7\" (UID: \"1eb42749-defc-4272-a671-4569a9705519\") " pod="openshift-monitoring/node-exporter-hcvz7" Jul 23 17:46:12.505467 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.505341 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1eb42749-defc-4272-a671-4569a9705519-root\") pod \"node-exporter-hcvz7\" (UID: \"1eb42749-defc-4272-a671-4569a9705519\") " pod="openshift-monitoring/node-exporter-hcvz7" Jul 23 17:46:12.505467 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.505371 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1eb42749-defc-4272-a671-4569a9705519-node-exporter-tls\") pod \"node-exporter-hcvz7\" (UID: \"1eb42749-defc-4272-a671-4569a9705519\") " pod="openshift-monitoring/node-exporter-hcvz7" Jul 23 17:46:12.505467 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.505386 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1eb42749-defc-4272-a671-4569a9705519-node-exporter-wtmp\") pod \"node-exporter-hcvz7\" (UID: \"1eb42749-defc-4272-a671-4569a9705519\") " pod="openshift-monitoring/node-exporter-hcvz7" Jul 23 17:46:12.505467 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.505383 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1eb42749-defc-4272-a671-4569a9705519-sys\") pod \"node-exporter-hcvz7\" (UID: \"1eb42749-defc-4272-a671-4569a9705519\") " pod="openshift-monitoring/node-exporter-hcvz7" Jul 23 17:46:12.505467 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.505420 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1eb42749-defc-4272-a671-4569a9705519-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hcvz7\" (UID: \"1eb42749-defc-4272-a671-4569a9705519\") " pod="openshift-monitoring/node-exporter-hcvz7" Jul 23 17:46:12.505467 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.505455 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1eb42749-defc-4272-a671-4569a9705519-root\") pod \"node-exporter-hcvz7\" (UID: \"1eb42749-defc-4272-a671-4569a9705519\") " pod="openshift-monitoring/node-exporter-hcvz7" Jul 23 17:46:12.505467 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.505455 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1eb42749-defc-4272-a671-4569a9705519-node-exporter-accelerators-collector-config\") pod \"node-exporter-hcvz7\" (UID: \"1eb42749-defc-4272-a671-4569a9705519\") " pod="openshift-monitoring/node-exporter-hcvz7" Jul 23 17:46:12.505939 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.505771 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1eb42749-defc-4272-a671-4569a9705519-node-exporter-wtmp\") pod \"node-exporter-hcvz7\" (UID: \"1eb42749-defc-4272-a671-4569a9705519\") " pod="openshift-monitoring/node-exporter-hcvz7" Jul 23 17:46:12.505939 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.505820 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1eb42749-defc-4272-a671-4569a9705519-metrics-client-ca\") pod \"node-exporter-hcvz7\" (UID: \"1eb42749-defc-4272-a671-4569a9705519\") " pod="openshift-monitoring/node-exporter-hcvz7" Jul 23 17:46:12.505939 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.505874 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1eb42749-defc-4272-a671-4569a9705519-node-exporter-textfile\") pod \"node-exporter-hcvz7\" (UID: \"1eb42749-defc-4272-a671-4569a9705519\") " pod="openshift-monitoring/node-exporter-hcvz7" Jul 23 17:46:12.505939 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:46:12.505879 2565 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 23 17:46:12.506133 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:46:12.505955 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1eb42749-defc-4272-a671-4569a9705519-node-exporter-tls podName:1eb42749-defc-4272-a671-4569a9705519 nodeName:}" failed. No retries permitted until 2026-07-23 17:46:13.005934843 +0000 UTC m=+117.084238188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1eb42749-defc-4272-a671-4569a9705519-node-exporter-tls") pod "node-exporter-hcvz7" (UID: "1eb42749-defc-4272-a671-4569a9705519") : secret "node-exporter-tls" not found Jul 23 17:46:12.506133 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.505960 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1eb42749-defc-4272-a671-4569a9705519-node-exporter-accelerators-collector-config\") pod \"node-exporter-hcvz7\" (UID: \"1eb42749-defc-4272-a671-4569a9705519\") " pod="openshift-monitoring/node-exporter-hcvz7" Jul 23 17:46:12.508231 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.508203 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1eb42749-defc-4272-a671-4569a9705519-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hcvz7\" (UID: \"1eb42749-defc-4272-a671-4569a9705519\") " pod="openshift-monitoring/node-exporter-hcvz7" Jul 23 17:46:12.517945 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:12.517899 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnr2w\" (UniqueName: \"kubernetes.io/projected/1eb42749-defc-4272-a671-4569a9705519-kube-api-access-qnr2w\") pod \"node-exporter-hcvz7\" (UID: \"1eb42749-defc-4272-a671-4569a9705519\") " pod="openshift-monitoring/node-exporter-hcvz7" Jul 23 17:46:13.010923 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:13.010883 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1eb42749-defc-4272-a671-4569a9705519-node-exporter-tls\") pod \"node-exporter-hcvz7\" (UID: \"1eb42749-defc-4272-a671-4569a9705519\") " pod="openshift-monitoring/node-exporter-hcvz7" Jul 23 17:46:13.013616 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:13.013576 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1eb42749-defc-4272-a671-4569a9705519-node-exporter-tls\") pod \"node-exporter-hcvz7\" (UID: \"1eb42749-defc-4272-a671-4569a9705519\") " pod="openshift-monitoring/node-exporter-hcvz7" Jul 23 17:46:13.207546 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:13.207513 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hcvz7" Jul 23 17:46:13.836939 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:46:13.836905 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb42749_defc_4272_a671_4569a9705519.slice/crio-387db5e39d18b68edb0cbd087ee5f3a790307ee08667eba241e1b6aebb9b8eaa WatchSource:0}: Error finding container 387db5e39d18b68edb0cbd087ee5f3a790307ee08667eba241e1b6aebb9b8eaa: Status 404 returned error can't find the container with id 387db5e39d18b68edb0cbd087ee5f3a790307ee08667eba241e1b6aebb9b8eaa Jul 23 17:46:13.974327 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:13.974287 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vsw5b" event={"ID":"35c70c4c-f5f0-49a6-a61f-f8676deeb25a","Type":"ContainerStarted","Data":"5a06cabc3ea15b78ad4a9733a1df89af9502341269eea7c65186a1d844d6fac6"} Jul 23 17:46:13.976415 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:13.976388 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hcvz7" event={"ID":"1eb42749-defc-4272-a671-4569a9705519","Type":"ContainerStarted","Data":"387db5e39d18b68edb0cbd087ee5f3a790307ee08667eba241e1b6aebb9b8eaa"} Jul 23 17:46:14.713500 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:14.713451 2565 patch_prober.go:28] interesting pod/image-registry-dcf9785b6-pbsjf container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 23 17:46:14.713980 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:14.713523 2565 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" podUID="0aaf3783-02d1-4b26-a61d-b960d83d1b30" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 17:46:14.987211 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:14.987125 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-jrsv7" event={"ID":"6a7d3768-fb48-410a-b5cb-e936bdfb9baf","Type":"ContainerStarted","Data":"b62c513edc54281d3e1008617129c546fe52518df2f5685b6056031b951477ee"} Jul 23 17:46:14.987446 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:14.987428 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-jrsv7" Jul 23 17:46:14.989839 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:14.989324 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff97bc65f-5fvcc" event={"ID":"6a9979ef-2936-4c14-8ef1-2b0393f5abcc","Type":"ContainerStarted","Data":"fb83ac38ca20cd89d0c2dcdd6d5db7436156dd1f2a89986a18af53bc7932c60e"} Jul 23 17:46:14.989839 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:14.989402 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vsw5b" Jul 23 17:46:15.001348 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:15.001316 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-jrsv7" Jul 23 17:46:15.006977 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:15.006921 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-jrsv7" podStartSLOduration=2.421778637 podStartE2EDuration="24.006904892s" totalImagesPullingTime="21.585126255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:45:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:46:15.005378792 +0000 UTC m=+119.083682167" watchObservedRunningTime="2026-07-23 17:46:15.006904892 +0000 UTC m=+119.085208244" Jul 23 17:46:15.027619 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:15.027541 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vsw5b" podStartSLOduration=83.030596012 podStartE2EDuration="1m25.027522904s" totalImagesPullingTime="1.996926892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:46:15.025153963 +0000 UTC m=+119.103457316" watchObservedRunningTime="2026-07-23 17:46:15.027522904 +0000 UTC m=+119.105826256" Jul 23 17:46:15.082056 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:15.081998 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-ff97bc65f-5fvcc" podStartSLOduration=7.163765483 podStartE2EDuration="18.081972996s" totalImagesPullingTime="10.918207513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:45:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:46:15.051440798 +0000 UTC m=+119.129744153" watchObservedRunningTime="2026-07-23 17:46:15.081972996 +0000 UTC m=+119.160276349" Jul 23 17:46:15.997816 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:15.997779 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vsw5b" Jul 23 17:46:16.880448 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:16.879529 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:46:16.970471 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:16.970434 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ff97bc65f-5fvcc"] Jul 23 17:46:17.002704 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:17.002664 2565 generic.go:356] "Generic (PLEG): container finished" podID="1eb42749-defc-4272-a671-4569a9705519" containerID="456feba7ab3e6b82aff519aa15b767e4030ea387b366fae10ca186e63b1d1e47" exitCode=0 Jul 23 17:46:17.003752 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:17.003722 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hcvz7" event={"ID":"1eb42749-defc-4272-a671-4569a9705519","Type":"ContainerDied","Data":"456feba7ab3e6b82aff519aa15b767e4030ea387b366fae10ca186e63b1d1e47"} Jul 23 17:46:17.400646 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:17.400570 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-ff97bc65f-5fvcc" Jul 23 17:46:18.009935 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:18.009888 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hcvz7" event={"ID":"1eb42749-defc-4272-a671-4569a9705519","Type":"ContainerStarted","Data":"272e9107cdeab8efe08e7531b4e3d3510eb69df62256d6b4bae6e0c8c899c546"} Jul 23 17:46:18.010353 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:18.009950 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hcvz7" event={"ID":"1eb42749-defc-4272-a671-4569a9705519","Type":"ContainerStarted","Data":"193d472acb4ed6f8b6db2a6e538f92b498de44600a936d889db0268300917d26"} Jul 23 17:46:18.040588 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:18.040532 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hcvz7" podStartSLOduration=3.606870969 podStartE2EDuration="6.040515573s" totalImagesPullingTime="2.336702752s" totalInitContainerRuntime="96.941852ms" isStatefulPod=true podCreationTimestamp="2026-07-23 17:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:46:18.038142084 +0000 UTC m=+122.116445438" watchObservedRunningTime="2026-07-23 17:46:18.040515573 +0000 UTC m=+122.118818925" Jul 23 17:46:19.712162 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:19.712131 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-dcf9785b6-pbsjf"] Jul 23 17:46:21.602164 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:21.602127 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6fcbb68d96-sj5bs"] Jul 23 17:46:21.690312 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:21.690276 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fcbb68d96-sj5bs"] Jul 23 17:46:21.690491 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:21.690389 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fcbb68d96-sj5bs" Jul 23 17:46:21.698898 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:21.698551 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 23 17:46:21.795436 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:21.795399 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6d63f929-fa89-4593-8043-ca32b2f1fa81-console-config\") pod \"console-6fcbb68d96-sj5bs\" (UID: \"6d63f929-fa89-4593-8043-ca32b2f1fa81\") " pod="openshift-console/console-6fcbb68d96-sj5bs" Jul 23 17:46:21.795651 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:21.795447 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6d63f929-fa89-4593-8043-ca32b2f1fa81-console-serving-cert\") pod \"console-6fcbb68d96-sj5bs\" (UID: \"6d63f929-fa89-4593-8043-ca32b2f1fa81\") " pod="openshift-console/console-6fcbb68d96-sj5bs" Jul 23 17:46:21.795651 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:21.795484 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6d63f929-fa89-4593-8043-ca32b2f1fa81-console-oauth-config\") pod \"console-6fcbb68d96-sj5bs\" (UID: \"6d63f929-fa89-4593-8043-ca32b2f1fa81\") " pod="openshift-console/console-6fcbb68d96-sj5bs" Jul 23 17:46:21.795651 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:21.795550 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwmmd\" (UniqueName: \"kubernetes.io/projected/6d63f929-fa89-4593-8043-ca32b2f1fa81-kube-api-access-fwmmd\") pod \"console-6fcbb68d96-sj5bs\" (UID: \"6d63f929-fa89-4593-8043-ca32b2f1fa81\") " pod="openshift-console/console-6fcbb68d96-sj5bs" Jul 23 17:46:21.795651 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:21.795583 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6d63f929-fa89-4593-8043-ca32b2f1fa81-service-ca\") pod \"console-6fcbb68d96-sj5bs\" (UID: \"6d63f929-fa89-4593-8043-ca32b2f1fa81\") " pod="openshift-console/console-6fcbb68d96-sj5bs" Jul 23 17:46:21.795835 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:21.795649 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d63f929-fa89-4593-8043-ca32b2f1fa81-trusted-ca-bundle\") pod \"console-6fcbb68d96-sj5bs\" (UID: \"6d63f929-fa89-4593-8043-ca32b2f1fa81\") " pod="openshift-console/console-6fcbb68d96-sj5bs" Jul 23 17:46:21.795835 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:21.795732 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6d63f929-fa89-4593-8043-ca32b2f1fa81-oauth-serving-cert\") pod \"console-6fcbb68d96-sj5bs\" (UID: \"6d63f929-fa89-4593-8043-ca32b2f1fa81\") " pod="openshift-console/console-6fcbb68d96-sj5bs" Jul 23 17:46:21.896559 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:21.896469 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwmmd\" (UniqueName: \"kubernetes.io/projected/6d63f929-fa89-4593-8043-ca32b2f1fa81-kube-api-access-fwmmd\") pod \"console-6fcbb68d96-sj5bs\" (UID: \"6d63f929-fa89-4593-8043-ca32b2f1fa81\") " pod="openshift-console/console-6fcbb68d96-sj5bs" Jul 23 17:46:21.896559 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:21.896522 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6d63f929-fa89-4593-8043-ca32b2f1fa81-service-ca\") pod \"console-6fcbb68d96-sj5bs\" (UID: \"6d63f929-fa89-4593-8043-ca32b2f1fa81\") " pod="openshift-console/console-6fcbb68d96-sj5bs" Jul 23 17:46:21.896836 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:21.896639 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d63f929-fa89-4593-8043-ca32b2f1fa81-trusted-ca-bundle\") pod \"console-6fcbb68d96-sj5bs\" (UID: \"6d63f929-fa89-4593-8043-ca32b2f1fa81\") " pod="openshift-console/console-6fcbb68d96-sj5bs" Jul 23 17:46:21.896836 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:21.896709 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6d63f929-fa89-4593-8043-ca32b2f1fa81-oauth-serving-cert\") pod \"console-6fcbb68d96-sj5bs\" (UID: \"6d63f929-fa89-4593-8043-ca32b2f1fa81\") " pod="openshift-console/console-6fcbb68d96-sj5bs" Jul 23 17:46:21.896836 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:21.896776 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6d63f929-fa89-4593-8043-ca32b2f1fa81-console-config\") pod \"console-6fcbb68d96-sj5bs\" (UID: \"6d63f929-fa89-4593-8043-ca32b2f1fa81\") " pod="openshift-console/console-6fcbb68d96-sj5bs" Jul 23 17:46:21.896836 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:21.896806 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6d63f929-fa89-4593-8043-ca32b2f1fa81-console-serving-cert\") pod \"console-6fcbb68d96-sj5bs\" (UID: \"6d63f929-fa89-4593-8043-ca32b2f1fa81\") " pod="openshift-console/console-6fcbb68d96-sj5bs" Jul 23 17:46:21.897038 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:21.896843 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6d63f929-fa89-4593-8043-ca32b2f1fa81-console-oauth-config\") pod \"console-6fcbb68d96-sj5bs\" (UID: \"6d63f929-fa89-4593-8043-ca32b2f1fa81\") " pod="openshift-console/console-6fcbb68d96-sj5bs" Jul 23 17:46:21.897352 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:21.897323 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6d63f929-fa89-4593-8043-ca32b2f1fa81-service-ca\") pod \"console-6fcbb68d96-sj5bs\" (UID: \"6d63f929-fa89-4593-8043-ca32b2f1fa81\") " pod="openshift-console/console-6fcbb68d96-sj5bs" Jul 23 17:46:21.897459 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:21.897327 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6d63f929-fa89-4593-8043-ca32b2f1fa81-oauth-serving-cert\") pod \"console-6fcbb68d96-sj5bs\" (UID: \"6d63f929-fa89-4593-8043-ca32b2f1fa81\") " pod="openshift-console/console-6fcbb68d96-sj5bs" Jul 23 17:46:21.898043 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:21.897992 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6d63f929-fa89-4593-8043-ca32b2f1fa81-console-config\") pod \"console-6fcbb68d96-sj5bs\" (UID: \"6d63f929-fa89-4593-8043-ca32b2f1fa81\") " pod="openshift-console/console-6fcbb68d96-sj5bs" Jul 23 17:46:21.898171 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:21.898112 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d63f929-fa89-4593-8043-ca32b2f1fa81-trusted-ca-bundle\") pod \"console-6fcbb68d96-sj5bs\" (UID: \"6d63f929-fa89-4593-8043-ca32b2f1fa81\") " pod="openshift-console/console-6fcbb68d96-sj5bs" Jul 23 17:46:21.899871 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:21.899849 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6d63f929-fa89-4593-8043-ca32b2f1fa81-console-oauth-config\") pod \"console-6fcbb68d96-sj5bs\" (UID: \"6d63f929-fa89-4593-8043-ca32b2f1fa81\") " pod="openshift-console/console-6fcbb68d96-sj5bs" Jul 23 17:46:21.900074 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:21.900050 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6d63f929-fa89-4593-8043-ca32b2f1fa81-console-serving-cert\") pod \"console-6fcbb68d96-sj5bs\" (UID: \"6d63f929-fa89-4593-8043-ca32b2f1fa81\") " pod="openshift-console/console-6fcbb68d96-sj5bs" Jul 23 17:46:21.906376 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:21.906351 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwmmd\" (UniqueName: \"kubernetes.io/projected/6d63f929-fa89-4593-8043-ca32b2f1fa81-kube-api-access-fwmmd\") pod \"console-6fcbb68d96-sj5bs\" (UID: \"6d63f929-fa89-4593-8043-ca32b2f1fa81\") " pod="openshift-console/console-6fcbb68d96-sj5bs" Jul 23 17:46:22.002657 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:22.002620 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fcbb68d96-sj5bs" Jul 23 17:46:22.187652 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:22.187616 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fcbb68d96-sj5bs"] Jul 23 17:46:22.190835 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:46:22.190807 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d63f929_fa89_4593_8043_ca32b2f1fa81.slice/crio-530263eab6ee8ab62ac4c53070a6735ba72da6d9976ebca6fa49b358bdb571ac WatchSource:0}: Error finding container 530263eab6ee8ab62ac4c53070a6735ba72da6d9976ebca6fa49b358bdb571ac: Status 404 returned error can't find the container with id 530263eab6ee8ab62ac4c53070a6735ba72da6d9976ebca6fa49b358bdb571ac Jul 23 17:46:23.035845 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:23.035808 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fcbb68d96-sj5bs" event={"ID":"6d63f929-fa89-4593-8043-ca32b2f1fa81","Type":"ContainerStarted","Data":"6f87883940ef4b0e76caa62375f7c594c262f3eb1364d50d6848d0e17937f22b"} Jul 23 17:46:23.036305 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:23.035872 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fcbb68d96-sj5bs" event={"ID":"6d63f929-fa89-4593-8043-ca32b2f1fa81","Type":"ContainerStarted","Data":"530263eab6ee8ab62ac4c53070a6735ba72da6d9976ebca6fa49b358bdb571ac"} Jul 23 17:46:23.058682 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:23.058630 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6fcbb68d96-sj5bs" podStartSLOduration=2.058615414 podStartE2EDuration="2.058615414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:46:23.057130408 +0000 UTC m=+127.135433762" watchObservedRunningTime="2026-07-23 17:46:23.058615414 +0000 UTC m=+127.136918763" Jul 23 17:46:26.240108 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:26.240069 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7027def-1a58-4d31-a9ce-ea9814a550a3-metrics-certs\") pod \"network-metrics-daemon-t2hlf\" (UID: \"d7027def-1a58-4d31-a9ce-ea9814a550a3\") " pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:46:26.242374 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:26.242353 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7027def-1a58-4d31-a9ce-ea9814a550a3-metrics-certs\") pod \"network-metrics-daemon-t2hlf\" (UID: \"d7027def-1a58-4d31-a9ce-ea9814a550a3\") " pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:46:26.527001 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:26.526973 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2xnzn\"" Jul 23 17:46:26.535039 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:26.535013 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2hlf" Jul 23 17:46:26.704862 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:26.704812 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t2hlf"] Jul 23 17:46:26.707073 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:46:26.707041 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7027def_1a58_4d31_a9ce_ea9814a550a3.slice/crio-5a04c1c71cf3d99d9b178c2250a143ea0f4f7a9bfb457cc40dd6a8bb0a443d57 WatchSource:0}: Error finding container 5a04c1c71cf3d99d9b178c2250a143ea0f4f7a9bfb457cc40dd6a8bb0a443d57: Status 404 returned error can't find the container with id 5a04c1c71cf3d99d9b178c2250a143ea0f4f7a9bfb457cc40dd6a8bb0a443d57 Jul 23 17:46:27.052859 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:27.052825 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t2hlf" event={"ID":"d7027def-1a58-4d31-a9ce-ea9814a550a3","Type":"ContainerStarted","Data":"5a04c1c71cf3d99d9b178c2250a143ea0f4f7a9bfb457cc40dd6a8bb0a443d57"} Jul 23 17:46:29.063387 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:29.063348 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t2hlf" event={"ID":"d7027def-1a58-4d31-a9ce-ea9814a550a3","Type":"ContainerStarted","Data":"501f98528bd52ea68ebc6d399c6b20401f2dc2a103fa72f81b66e71e7866c626"} Jul 23 17:46:29.063840 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:29.063437 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t2hlf" event={"ID":"d7027def-1a58-4d31-a9ce-ea9814a550a3","Type":"ContainerStarted","Data":"3c101a89258088bb12f068be48d7d153cad33e6ea7239df3553a61456814ea2e"} Jul 23 17:46:29.083740 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:29.083690 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-t2hlf" podStartSLOduration=131.552275469 podStartE2EDuration="2m13.08367379s" totalImagesPullingTime="1.531398321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:44:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:46:29.082450944 +0000 UTC m=+133.160754297" watchObservedRunningTime="2026-07-23 17:46:29.08367379 +0000 UTC m=+133.161977168" Jul 23 17:46:32.003491 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:32.003453 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6fcbb68d96-sj5bs" Jul 23 17:46:32.003939 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:32.003530 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6fcbb68d96-sj5bs" Jul 23 17:46:32.008438 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:32.008413 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6fcbb68d96-sj5bs" Jul 23 17:46:32.082007 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:32.081977 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6fcbb68d96-sj5bs" Jul 23 17:46:42.030226 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:42.030165 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-ff97bc65f-5fvcc" podUID="6a9979ef-2936-4c14-8ef1-2b0393f5abcc" containerName="console" containerID="cri-o://fb83ac38ca20cd89d0c2dcdd6d5db7436156dd1f2a89986a18af53bc7932c60e" gracePeriod=15 Jul 23 17:46:42.289142 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:42.289085 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff97bc65f-5fvcc" Jul 23 17:46:42.381929 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:42.381896 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-oauth-serving-cert\") pod \"6a9979ef-2936-4c14-8ef1-2b0393f5abcc\" (UID: \"6a9979ef-2936-4c14-8ef1-2b0393f5abcc\") " Jul 23 17:46:42.382092 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:42.381993 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-console-config\") pod \"6a9979ef-2936-4c14-8ef1-2b0393f5abcc\" (UID: \"6a9979ef-2936-4c14-8ef1-2b0393f5abcc\") " Jul 23 17:46:42.382092 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:42.382017 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fj4n\" (UniqueName: \"kubernetes.io/projected/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-kube-api-access-8fj4n\") pod \"6a9979ef-2936-4c14-8ef1-2b0393f5abcc\" (UID: \"6a9979ef-2936-4c14-8ef1-2b0393f5abcc\") " Jul 23 17:46:42.382092 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:42.382033 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-service-ca\") pod \"6a9979ef-2936-4c14-8ef1-2b0393f5abcc\" (UID: \"6a9979ef-2936-4c14-8ef1-2b0393f5abcc\") " Jul 23 17:46:42.382092 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:42.382066 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-console-serving-cert\") pod \"6a9979ef-2936-4c14-8ef1-2b0393f5abcc\" (UID: \"6a9979ef-2936-4c14-8ef1-2b0393f5abcc\") " Jul 23 17:46:42.382296 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:42.382117 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-console-oauth-config\") pod \"6a9979ef-2936-4c14-8ef1-2b0393f5abcc\" (UID: \"6a9979ef-2936-4c14-8ef1-2b0393f5abcc\") " Jul 23 17:46:42.382479 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:42.382442 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6a9979ef-2936-4c14-8ef1-2b0393f5abcc" (UID: "6a9979ef-2936-4c14-8ef1-2b0393f5abcc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 17:46:42.382479 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:42.382464 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-console-config" (OuterVolumeSpecName: "console-config") pod "6a9979ef-2936-4c14-8ef1-2b0393f5abcc" (UID: "6a9979ef-2936-4c14-8ef1-2b0393f5abcc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 17:46:42.382479 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:42.382471 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-service-ca" (OuterVolumeSpecName: "service-ca") pod "6a9979ef-2936-4c14-8ef1-2b0393f5abcc" (UID: "6a9979ef-2936-4c14-8ef1-2b0393f5abcc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 17:46:42.384416 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:42.384388 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6a9979ef-2936-4c14-8ef1-2b0393f5abcc" (UID: "6a9979ef-2936-4c14-8ef1-2b0393f5abcc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 17:46:42.384516 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:42.384415 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6a9979ef-2936-4c14-8ef1-2b0393f5abcc" (UID: "6a9979ef-2936-4c14-8ef1-2b0393f5abcc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 17:46:42.384516 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:42.384449 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-kube-api-access-8fj4n" (OuterVolumeSpecName: "kube-api-access-8fj4n") pod "6a9979ef-2936-4c14-8ef1-2b0393f5abcc" (UID: "6a9979ef-2936-4c14-8ef1-2b0393f5abcc"). InnerVolumeSpecName "kube-api-access-8fj4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 17:46:42.483237 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:42.483194 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-console-oauth-config\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 17:46:42.483237 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:42.483224 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-oauth-serving-cert\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 17:46:42.483237 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:42.483234 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-console-config\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 17:46:42.483237 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:42.483243 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fj4n\" (UniqueName: \"kubernetes.io/projected/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-kube-api-access-8fj4n\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 17:46:42.483486 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:42.483253 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-service-ca\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 17:46:42.483486 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:42.483262 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a9979ef-2936-4c14-8ef1-2b0393f5abcc-console-serving-cert\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 17:46:43.130896 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:43.130852 2565 generic.go:356] "Generic (PLEG): container finished" podID="6a9979ef-2936-4c14-8ef1-2b0393f5abcc" containerID="fb83ac38ca20cd89d0c2dcdd6d5db7436156dd1f2a89986a18af53bc7932c60e" exitCode=0 Jul 23 17:46:43.130896 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:43.130896 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff97bc65f-5fvcc" Jul 23 17:46:43.131325 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:43.130900 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff97bc65f-5fvcc" event={"ID":"6a9979ef-2936-4c14-8ef1-2b0393f5abcc","Type":"ContainerDied","Data":"fb83ac38ca20cd89d0c2dcdd6d5db7436156dd1f2a89986a18af53bc7932c60e"} Jul 23 17:46:43.131325 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:43.130955 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff97bc65f-5fvcc" event={"ID":"6a9979ef-2936-4c14-8ef1-2b0393f5abcc","Type":"ContainerDied","Data":"5b157663c4bf11c56cba7d173a5c2562b62d48806cf523d9438b1a99619987a3"} Jul 23 17:46:43.131325 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:43.130975 2565 scope.go:117] "RemoveContainer" containerID="fb83ac38ca20cd89d0c2dcdd6d5db7436156dd1f2a89986a18af53bc7932c60e" Jul 23 17:46:43.139251 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:43.139227 2565 scope.go:117] "RemoveContainer" containerID="fb83ac38ca20cd89d0c2dcdd6d5db7436156dd1f2a89986a18af53bc7932c60e" Jul 23 17:46:43.139513 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:46:43.139491 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb83ac38ca20cd89d0c2dcdd6d5db7436156dd1f2a89986a18af53bc7932c60e\": container with ID starting with fb83ac38ca20cd89d0c2dcdd6d5db7436156dd1f2a89986a18af53bc7932c60e not found: ID does not exist" containerID="fb83ac38ca20cd89d0c2dcdd6d5db7436156dd1f2a89986a18af53bc7932c60e" Jul 23 17:46:43.139585 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:43.139524 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb83ac38ca20cd89d0c2dcdd6d5db7436156dd1f2a89986a18af53bc7932c60e"} err="failed to get container status \"fb83ac38ca20cd89d0c2dcdd6d5db7436156dd1f2a89986a18af53bc7932c60e\": rpc error: code = NotFound desc = could not find container \"fb83ac38ca20cd89d0c2dcdd6d5db7436156dd1f2a89986a18af53bc7932c60e\": container with ID starting with fb83ac38ca20cd89d0c2dcdd6d5db7436156dd1f2a89986a18af53bc7932c60e not found: ID does not exist" Jul 23 17:46:43.148428 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:43.148402 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ff97bc65f-5fvcc"] Jul 23 17:46:43.152652 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:43.152630 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-ff97bc65f-5fvcc"] Jul 23 17:46:44.405517 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:44.405485 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a9979ef-2936-4c14-8ef1-2b0393f5abcc" path="/var/lib/kubelet/pods/6a9979ef-2936-4c14-8ef1-2b0393f5abcc/volumes" Jul 23 17:46:44.736075 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:44.736029 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" podUID="0aaf3783-02d1-4b26-a61d-b960d83d1b30" containerName="registry" containerID="cri-o://f2527a09269fded234580c736f663fa8c4fa47002594faaeb9c8554a9d295213" gracePeriod=30 Jul 23 17:46:44.984017 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:44.983989 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:46:45.105174 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.105139 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-bound-sa-token\") pod \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " Jul 23 17:46:45.105365 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.105187 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-tls\") pod \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " Jul 23 17:46:45.105365 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.105209 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0aaf3783-02d1-4b26-a61d-b960d83d1b30-trusted-ca\") pod \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " Jul 23 17:46:45.105365 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.105247 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0aaf3783-02d1-4b26-a61d-b960d83d1b30-ca-trust-extracted\") pod \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " Jul 23 17:46:45.105365 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.105302 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pn8d9\" (UniqueName: \"kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-kube-api-access-pn8d9\") pod \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " Jul 23 17:46:45.105365 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.105331 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0aaf3783-02d1-4b26-a61d-b960d83d1b30-image-registry-private-configuration\") pod \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " Jul 23 17:46:45.105365 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.105365 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0aaf3783-02d1-4b26-a61d-b960d83d1b30-installation-pull-secrets\") pod \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " Jul 23 17:46:45.105714 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.105399 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-certificates\") pod \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\" (UID: \"0aaf3783-02d1-4b26-a61d-b960d83d1b30\") " Jul 23 17:46:45.105810 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.105781 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aaf3783-02d1-4b26-a61d-b960d83d1b30-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "0aaf3783-02d1-4b26-a61d-b960d83d1b30" (UID: "0aaf3783-02d1-4b26-a61d-b960d83d1b30"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 17:46:45.106234 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.106193 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "0aaf3783-02d1-4b26-a61d-b960d83d1b30" (UID: "0aaf3783-02d1-4b26-a61d-b960d83d1b30"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 17:46:45.107940 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.107912 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0aaf3783-02d1-4b26-a61d-b960d83d1b30" (UID: "0aaf3783-02d1-4b26-a61d-b960d83d1b30"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 17:46:45.108159 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.108140 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "0aaf3783-02d1-4b26-a61d-b960d83d1b30" (UID: "0aaf3783-02d1-4b26-a61d-b960d83d1b30"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 17:46:45.108634 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.108590 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aaf3783-02d1-4b26-a61d-b960d83d1b30-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "0aaf3783-02d1-4b26-a61d-b960d83d1b30" (UID: "0aaf3783-02d1-4b26-a61d-b960d83d1b30"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 17:46:45.108749 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.108693 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aaf3783-02d1-4b26-a61d-b960d83d1b30-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "0aaf3783-02d1-4b26-a61d-b960d83d1b30" (UID: "0aaf3783-02d1-4b26-a61d-b960d83d1b30"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 17:46:45.108749 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.108736 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-kube-api-access-pn8d9" (OuterVolumeSpecName: "kube-api-access-pn8d9") pod "0aaf3783-02d1-4b26-a61d-b960d83d1b30" (UID: "0aaf3783-02d1-4b26-a61d-b960d83d1b30"). InnerVolumeSpecName "kube-api-access-pn8d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 17:46:45.113582 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.113556 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aaf3783-02d1-4b26-a61d-b960d83d1b30-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "0aaf3783-02d1-4b26-a61d-b960d83d1b30" (UID: "0aaf3783-02d1-4b26-a61d-b960d83d1b30"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 17:46:45.141725 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.141694 2565 generic.go:356] "Generic (PLEG): container finished" podID="9b61f887-8f31-4935-9a55-626c4d3671e8" containerID="4874a01054b402cfd38ddaea153c98d3cd782db1f0e64733bfe8615ed5947a72" exitCode=0 Jul 23 17:46:45.141861 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.141769 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-xttxr" event={"ID":"9b61f887-8f31-4935-9a55-626c4d3671e8","Type":"ContainerDied","Data":"4874a01054b402cfd38ddaea153c98d3cd782db1f0e64733bfe8615ed5947a72"} Jul 23 17:46:45.142161 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.142145 2565 scope.go:117] "RemoveContainer" containerID="4874a01054b402cfd38ddaea153c98d3cd782db1f0e64733bfe8615ed5947a72" Jul 23 17:46:45.142930 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.142906 2565 generic.go:356] "Generic (PLEG): container finished" podID="0aaf3783-02d1-4b26-a61d-b960d83d1b30" containerID="f2527a09269fded234580c736f663fa8c4fa47002594faaeb9c8554a9d295213" exitCode=0 Jul 23 17:46:45.143014 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.142947 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" Jul 23 17:46:45.143014 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.142975 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" event={"ID":"0aaf3783-02d1-4b26-a61d-b960d83d1b30","Type":"ContainerDied","Data":"f2527a09269fded234580c736f663fa8c4fa47002594faaeb9c8554a9d295213"} Jul 23 17:46:45.143014 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.143012 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dcf9785b6-pbsjf" event={"ID":"0aaf3783-02d1-4b26-a61d-b960d83d1b30","Type":"ContainerDied","Data":"5966a4d67c4e805a1910902a5d9349a21fa9c1d6555a7030776ed82ad8cee346"} Jul 23 17:46:45.143129 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.143031 2565 scope.go:117] "RemoveContainer" containerID="f2527a09269fded234580c736f663fa8c4fa47002594faaeb9c8554a9d295213" Jul 23 17:46:45.154271 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.154251 2565 scope.go:117] "RemoveContainer" containerID="f2527a09269fded234580c736f663fa8c4fa47002594faaeb9c8554a9d295213" Jul 23 17:46:45.154511 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:46:45.154493 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2527a09269fded234580c736f663fa8c4fa47002594faaeb9c8554a9d295213\": container with ID starting with f2527a09269fded234580c736f663fa8c4fa47002594faaeb9c8554a9d295213 not found: ID does not exist" containerID="f2527a09269fded234580c736f663fa8c4fa47002594faaeb9c8554a9d295213" Jul 23 17:46:45.154565 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.154519 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2527a09269fded234580c736f663fa8c4fa47002594faaeb9c8554a9d295213"} err="failed to get container status \"f2527a09269fded234580c736f663fa8c4fa47002594faaeb9c8554a9d295213\": rpc error: code = NotFound desc = could not find container \"f2527a09269fded234580c736f663fa8c4fa47002594faaeb9c8554a9d295213\": container with ID starting with f2527a09269fded234580c736f663fa8c4fa47002594faaeb9c8554a9d295213 not found: ID does not exist" Jul 23 17:46:45.177880 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.177850 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-dcf9785b6-pbsjf"] Jul 23 17:46:45.183685 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.183652 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-dcf9785b6-pbsjf"] Jul 23 17:46:45.206752 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.206725 2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-tls\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 17:46:45.206752 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.206747 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0aaf3783-02d1-4b26-a61d-b960d83d1b30-trusted-ca\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 17:46:45.206905 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.206759 2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0aaf3783-02d1-4b26-a61d-b960d83d1b30-ca-trust-extracted\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 17:46:45.206905 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.206771 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn8d9\" (UniqueName: \"kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-kube-api-access-pn8d9\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 17:46:45.206905 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.206786 2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0aaf3783-02d1-4b26-a61d-b960d83d1b30-image-registry-private-configuration\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 17:46:45.206905 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.206798 2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0aaf3783-02d1-4b26-a61d-b960d83d1b30-installation-pull-secrets\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 17:46:45.206905 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.206810 2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0aaf3783-02d1-4b26-a61d-b960d83d1b30-registry-certificates\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 17:46:45.206905 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:45.206822 2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0aaf3783-02d1-4b26-a61d-b960d83d1b30-bound-sa-token\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 17:46:46.148906 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:46.148866 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-xttxr" event={"ID":"9b61f887-8f31-4935-9a55-626c4d3671e8","Type":"ContainerStarted","Data":"2e7136e884d800d88181c171e34fc7cfbc039c268b25c241f5081a084efb1ccd"} Jul 23 17:46:46.405646 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:46.405555 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0aaf3783-02d1-4b26-a61d-b960d83d1b30" path="/var/lib/kubelet/pods/0aaf3783-02d1-4b26-a61d-b960d83d1b30/volumes" Jul 23 17:46:49.163799 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:49.163762 2565 generic.go:356] "Generic (PLEG): container finished" podID="f590b7bd-8791-41a3-a520-9ba3559d3371" containerID="116d7332ee3586f861bc866387197e069ee29d69602454523f43f49f3c2a4083" exitCode=0 Jul 23 17:46:49.164298 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:49.163843 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-n55cw" event={"ID":"f590b7bd-8791-41a3-a520-9ba3559d3371","Type":"ContainerDied","Data":"116d7332ee3586f861bc866387197e069ee29d69602454523f43f49f3c2a4083"} Jul 23 17:46:49.164298 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:49.164293 2565 scope.go:117] "RemoveContainer" containerID="116d7332ee3586f861bc866387197e069ee29d69602454523f43f49f3c2a4083" Jul 23 17:46:50.170152 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:50.170113 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-n55cw" event={"ID":"f590b7bd-8791-41a3-a520-9ba3559d3371","Type":"ContainerStarted","Data":"8c9763c0160e905a1fc316ab8c221ebde61dfb192d7298f0b2a053be20b28d8e"} Jul 23 17:46:55.193813 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:55.193778 2565 generic.go:356] "Generic (PLEG): container finished" podID="e372e1b3-c8b6-4995-a770-6a970cd7f11e" containerID="f272ea3d872e264540b89847a974ab1fbfb2d7968943e1c98fed93d65afcf623" exitCode=0 Jul 23 17:46:55.194259 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:55.193855 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xh686" event={"ID":"e372e1b3-c8b6-4995-a770-6a970cd7f11e","Type":"ContainerDied","Data":"f272ea3d872e264540b89847a974ab1fbfb2d7968943e1c98fed93d65afcf623"} Jul 23 17:46:55.194259 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:55.194241 2565 scope.go:117] "RemoveContainer" containerID="f272ea3d872e264540b89847a974ab1fbfb2d7968943e1c98fed93d65afcf623" Jul 23 17:46:56.200415 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:46:56.200383 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-xh686" event={"ID":"e372e1b3-c8b6-4995-a770-6a970cd7f11e","Type":"ContainerStarted","Data":"5895291139c38c59322a1bd11a15b7d708b1a6b543677d68f17ee10673b32b59"} Jul 23 17:47:40.574775 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:47:40.574729 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fcbb68d96-sj5bs"] Jul 23 17:48:05.595348 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:05.595284 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6fcbb68d96-sj5bs" podUID="6d63f929-fa89-4593-8043-ca32b2f1fa81" containerName="console" containerID="cri-o://6f87883940ef4b0e76caa62375f7c594c262f3eb1364d50d6848d0e17937f22b" gracePeriod=15 Jul 23 17:48:05.839905 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:05.839878 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fcbb68d96-sj5bs" Jul 23 17:48:05.984327 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:05.984241 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6d63f929-fa89-4593-8043-ca32b2f1fa81-console-serving-cert\") pod \"6d63f929-fa89-4593-8043-ca32b2f1fa81\" (UID: \"6d63f929-fa89-4593-8043-ca32b2f1fa81\") " Jul 23 17:48:05.984327 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:05.984285 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6d63f929-fa89-4593-8043-ca32b2f1fa81-oauth-serving-cert\") pod \"6d63f929-fa89-4593-8043-ca32b2f1fa81\" (UID: \"6d63f929-fa89-4593-8043-ca32b2f1fa81\") " Jul 23 17:48:05.984327 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:05.984318 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwmmd\" (UniqueName: \"kubernetes.io/projected/6d63f929-fa89-4593-8043-ca32b2f1fa81-kube-api-access-fwmmd\") pod \"6d63f929-fa89-4593-8043-ca32b2f1fa81\" (UID: \"6d63f929-fa89-4593-8043-ca32b2f1fa81\") " Jul 23 17:48:05.984633 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:05.984341 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d63f929-fa89-4593-8043-ca32b2f1fa81-trusted-ca-bundle\") pod \"6d63f929-fa89-4593-8043-ca32b2f1fa81\" (UID: \"6d63f929-fa89-4593-8043-ca32b2f1fa81\") " Jul 23 17:48:05.984633 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:05.984491 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6d63f929-fa89-4593-8043-ca32b2f1fa81-service-ca\") pod \"6d63f929-fa89-4593-8043-ca32b2f1fa81\" (UID: \"6d63f929-fa89-4593-8043-ca32b2f1fa81\") " Jul 23 17:48:05.984633 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:05.984533 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6d63f929-fa89-4593-8043-ca32b2f1fa81-console-config\") pod \"6d63f929-fa89-4593-8043-ca32b2f1fa81\" (UID: \"6d63f929-fa89-4593-8043-ca32b2f1fa81\") " Jul 23 17:48:05.984633 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:05.984617 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6d63f929-fa89-4593-8043-ca32b2f1fa81-console-oauth-config\") pod \"6d63f929-fa89-4593-8043-ca32b2f1fa81\" (UID: \"6d63f929-fa89-4593-8043-ca32b2f1fa81\") " Jul 23 17:48:05.984838 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:05.984780 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d63f929-fa89-4593-8043-ca32b2f1fa81-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6d63f929-fa89-4593-8043-ca32b2f1fa81" (UID: "6d63f929-fa89-4593-8043-ca32b2f1fa81"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 17:48:05.984936 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:05.984908 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d63f929-fa89-4593-8043-ca32b2f1fa81-service-ca" (OuterVolumeSpecName: "service-ca") pod "6d63f929-fa89-4593-8043-ca32b2f1fa81" (UID: "6d63f929-fa89-4593-8043-ca32b2f1fa81"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 17:48:05.985000 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:05.984792 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d63f929-fa89-4593-8043-ca32b2f1fa81-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6d63f929-fa89-4593-8043-ca32b2f1fa81" (UID: "6d63f929-fa89-4593-8043-ca32b2f1fa81"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 17:48:05.985000 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:05.984974 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d63f929-fa89-4593-8043-ca32b2f1fa81-console-config" (OuterVolumeSpecName: "console-config") pod "6d63f929-fa89-4593-8043-ca32b2f1fa81" (UID: "6d63f929-fa89-4593-8043-ca32b2f1fa81"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 17:48:05.985078 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:05.985028 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6d63f929-fa89-4593-8043-ca32b2f1fa81-oauth-serving-cert\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 17:48:05.985078 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:05.985050 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d63f929-fa89-4593-8043-ca32b2f1fa81-trusted-ca-bundle\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 17:48:05.985078 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:05.985064 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6d63f929-fa89-4593-8043-ca32b2f1fa81-service-ca\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 17:48:05.986533 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:05.986500 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d63f929-fa89-4593-8043-ca32b2f1fa81-kube-api-access-fwmmd" (OuterVolumeSpecName: "kube-api-access-fwmmd") pod "6d63f929-fa89-4593-8043-ca32b2f1fa81" (UID: "6d63f929-fa89-4593-8043-ca32b2f1fa81"). InnerVolumeSpecName "kube-api-access-fwmmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 17:48:05.986533 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:05.986518 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d63f929-fa89-4593-8043-ca32b2f1fa81-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6d63f929-fa89-4593-8043-ca32b2f1fa81" (UID: "6d63f929-fa89-4593-8043-ca32b2f1fa81"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 17:48:05.986718 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:05.986542 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d63f929-fa89-4593-8043-ca32b2f1fa81-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6d63f929-fa89-4593-8043-ca32b2f1fa81" (UID: "6d63f929-fa89-4593-8043-ca32b2f1fa81"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 17:48:06.086244 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:06.086209 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6d63f929-fa89-4593-8043-ca32b2f1fa81-console-oauth-config\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 17:48:06.086244 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:06.086236 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6d63f929-fa89-4593-8043-ca32b2f1fa81-console-serving-cert\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 17:48:06.086244 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:06.086248 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwmmd\" (UniqueName: \"kubernetes.io/projected/6d63f929-fa89-4593-8043-ca32b2f1fa81-kube-api-access-fwmmd\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 17:48:06.086458 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:06.086260 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6d63f929-fa89-4593-8043-ca32b2f1fa81-console-config\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 17:48:06.494079 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:06.494047 2565 generic.go:356] "Generic (PLEG): container finished" podID="6d63f929-fa89-4593-8043-ca32b2f1fa81" containerID="6f87883940ef4b0e76caa62375f7c594c262f3eb1364d50d6848d0e17937f22b" exitCode=0 Jul 23 17:48:06.494275 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:06.494097 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fcbb68d96-sj5bs" event={"ID":"6d63f929-fa89-4593-8043-ca32b2f1fa81","Type":"ContainerDied","Data":"6f87883940ef4b0e76caa62375f7c594c262f3eb1364d50d6848d0e17937f22b"} Jul 23 17:48:06.494275 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:06.494101 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fcbb68d96-sj5bs" Jul 23 17:48:06.494275 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:06.494136 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fcbb68d96-sj5bs" event={"ID":"6d63f929-fa89-4593-8043-ca32b2f1fa81","Type":"ContainerDied","Data":"530263eab6ee8ab62ac4c53070a6735ba72da6d9976ebca6fa49b358bdb571ac"} Jul 23 17:48:06.494275 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:06.494154 2565 scope.go:117] "RemoveContainer" containerID="6f87883940ef4b0e76caa62375f7c594c262f3eb1364d50d6848d0e17937f22b" Jul 23 17:48:06.502285 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:06.502267 2565 scope.go:117] "RemoveContainer" containerID="6f87883940ef4b0e76caa62375f7c594c262f3eb1364d50d6848d0e17937f22b" Jul 23 17:48:06.502539 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:48:06.502522 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f87883940ef4b0e76caa62375f7c594c262f3eb1364d50d6848d0e17937f22b\": container with ID starting with 6f87883940ef4b0e76caa62375f7c594c262f3eb1364d50d6848d0e17937f22b not found: ID does not exist" containerID="6f87883940ef4b0e76caa62375f7c594c262f3eb1364d50d6848d0e17937f22b" Jul 23 17:48:06.502581 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:06.502549 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f87883940ef4b0e76caa62375f7c594c262f3eb1364d50d6848d0e17937f22b"} err="failed to get container status \"6f87883940ef4b0e76caa62375f7c594c262f3eb1364d50d6848d0e17937f22b\": rpc error: code = NotFound desc = could not find container \"6f87883940ef4b0e76caa62375f7c594c262f3eb1364d50d6848d0e17937f22b\": container with ID starting with 6f87883940ef4b0e76caa62375f7c594c262f3eb1364d50d6848d0e17937f22b not found: ID does not exist" Jul 23 17:48:06.511720 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:06.511690 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fcbb68d96-sj5bs"] Jul 23 17:48:06.514695 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:06.514675 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6fcbb68d96-sj5bs"] Jul 23 17:48:08.405014 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:48:08.404981 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d63f929-fa89-4593-8043-ca32b2f1fa81" path="/var/lib/kubelet/pods/6d63f929-fa89-4593-8043-ca32b2f1fa81/volumes" Jul 23 17:49:16.304851 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:49:16.304819 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-m7k4j_1cca3236-5c90-4abb-9b9b-39e40b1903c3/console-operator/1.log" Jul 23 17:49:16.307230 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:49:16.307205 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-m7k4j_1cca3236-5c90-4abb-9b9b-39e40b1903c3/console-operator/1.log" Jul 23 17:49:16.316670 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:49:16.316652 2565 kubelet.go:1648] "Image garbage collection succeeded" Jul 23 17:50:02.667374 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.667335 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-547c58d5d9-pfr5x"] Jul 23 17:50:02.667831 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.667700 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d63f929-fa89-4593-8043-ca32b2f1fa81" containerName="console" Jul 23 17:50:02.667831 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.667713 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d63f929-fa89-4593-8043-ca32b2f1fa81" containerName="console" Jul 23 17:50:02.667831 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.667726 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a9979ef-2936-4c14-8ef1-2b0393f5abcc" containerName="console" Jul 23 17:50:02.667831 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.667733 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a9979ef-2936-4c14-8ef1-2b0393f5abcc" containerName="console" Jul 23 17:50:02.667831 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.667741 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0aaf3783-02d1-4b26-a61d-b960d83d1b30" containerName="registry" Jul 23 17:50:02.667831 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.667747 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0aaf3783-02d1-4b26-a61d-b960d83d1b30" containerName="registry" Jul 23 17:50:02.667831 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.667817 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0aaf3783-02d1-4b26-a61d-b960d83d1b30" containerName="registry" Jul 23 17:50:02.667831 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.667830 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a9979ef-2936-4c14-8ef1-2b0393f5abcc" containerName="console" Jul 23 17:50:02.668068 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.667839 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d63f929-fa89-4593-8043-ca32b2f1fa81" containerName="console" Jul 23 17:50:02.670361 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.670344 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-547c58d5d9-pfr5x" Jul 23 17:50:02.673531 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.673508 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 23 17:50:02.673678 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.673638 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-25pnk\"" Jul 23 17:50:02.673805 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.673785 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 23 17:50:02.674282 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.674267 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 23 17:50:02.676259 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.676236 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 23 17:50:02.686203 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.686180 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-547c58d5d9-pfr5x"] Jul 23 17:50:02.812055 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.812017 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/740d1253-7e70-4b58-9c93-10cbd14d29d1-tmp\") pod \"opendatahub-operator-controller-manager-547c58d5d9-pfr5x\" (UID: \"740d1253-7e70-4b58-9c93-10cbd14d29d1\") " pod="opendatahub/opendatahub-operator-controller-manager-547c58d5d9-pfr5x" Jul 23 17:50:02.812238 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.812064 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54c7q\" (UniqueName: \"kubernetes.io/projected/740d1253-7e70-4b58-9c93-10cbd14d29d1-kube-api-access-54c7q\") pod \"opendatahub-operator-controller-manager-547c58d5d9-pfr5x\" (UID: \"740d1253-7e70-4b58-9c93-10cbd14d29d1\") " pod="opendatahub/opendatahub-operator-controller-manager-547c58d5d9-pfr5x" Jul 23 17:50:02.812238 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.812085 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/740d1253-7e70-4b58-9c93-10cbd14d29d1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-547c58d5d9-pfr5x\" (UID: \"740d1253-7e70-4b58-9c93-10cbd14d29d1\") " pod="opendatahub/opendatahub-operator-controller-manager-547c58d5d9-pfr5x" Jul 23 17:50:02.812238 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.812175 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/740d1253-7e70-4b58-9c93-10cbd14d29d1-manifests\") pod \"opendatahub-operator-controller-manager-547c58d5d9-pfr5x\" (UID: \"740d1253-7e70-4b58-9c93-10cbd14d29d1\") " pod="opendatahub/opendatahub-operator-controller-manager-547c58d5d9-pfr5x" Jul 23 17:50:02.812238 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.812220 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/740d1253-7e70-4b58-9c93-10cbd14d29d1-webhook-cert\") pod \"opendatahub-operator-controller-manager-547c58d5d9-pfr5x\" (UID: \"740d1253-7e70-4b58-9c93-10cbd14d29d1\") " pod="opendatahub/opendatahub-operator-controller-manager-547c58d5d9-pfr5x" Jul 23 17:50:02.912960 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.912917 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/740d1253-7e70-4b58-9c93-10cbd14d29d1-tmp\") pod \"opendatahub-operator-controller-manager-547c58d5d9-pfr5x\" (UID: \"740d1253-7e70-4b58-9c93-10cbd14d29d1\") " pod="opendatahub/opendatahub-operator-controller-manager-547c58d5d9-pfr5x" Jul 23 17:50:02.912960 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.912964 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54c7q\" (UniqueName: \"kubernetes.io/projected/740d1253-7e70-4b58-9c93-10cbd14d29d1-kube-api-access-54c7q\") pod \"opendatahub-operator-controller-manager-547c58d5d9-pfr5x\" (UID: \"740d1253-7e70-4b58-9c93-10cbd14d29d1\") " pod="opendatahub/opendatahub-operator-controller-manager-547c58d5d9-pfr5x" Jul 23 17:50:02.913215 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.912985 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/740d1253-7e70-4b58-9c93-10cbd14d29d1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-547c58d5d9-pfr5x\" (UID: \"740d1253-7e70-4b58-9c93-10cbd14d29d1\") " pod="opendatahub/opendatahub-operator-controller-manager-547c58d5d9-pfr5x" Jul 23 17:50:02.913215 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.913014 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/740d1253-7e70-4b58-9c93-10cbd14d29d1-manifests\") pod \"opendatahub-operator-controller-manager-547c58d5d9-pfr5x\" (UID: \"740d1253-7e70-4b58-9c93-10cbd14d29d1\") " pod="opendatahub/opendatahub-operator-controller-manager-547c58d5d9-pfr5x" Jul 23 17:50:02.913215 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.913044 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/740d1253-7e70-4b58-9c93-10cbd14d29d1-webhook-cert\") pod \"opendatahub-operator-controller-manager-547c58d5d9-pfr5x\" (UID: \"740d1253-7e70-4b58-9c93-10cbd14d29d1\") " pod="opendatahub/opendatahub-operator-controller-manager-547c58d5d9-pfr5x" Jul 23 17:50:02.913463 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.913423 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/740d1253-7e70-4b58-9c93-10cbd14d29d1-manifests\") pod \"opendatahub-operator-controller-manager-547c58d5d9-pfr5x\" (UID: \"740d1253-7e70-4b58-9c93-10cbd14d29d1\") " pod="opendatahub/opendatahub-operator-controller-manager-547c58d5d9-pfr5x" Jul 23 17:50:02.915232 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.915210 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/740d1253-7e70-4b58-9c93-10cbd14d29d1-tmp\") pod \"opendatahub-operator-controller-manager-547c58d5d9-pfr5x\" (UID: \"740d1253-7e70-4b58-9c93-10cbd14d29d1\") " pod="opendatahub/opendatahub-operator-controller-manager-547c58d5d9-pfr5x" Jul 23 17:50:02.915370 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.915351 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/740d1253-7e70-4b58-9c93-10cbd14d29d1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-547c58d5d9-pfr5x\" (UID: \"740d1253-7e70-4b58-9c93-10cbd14d29d1\") " pod="opendatahub/opendatahub-operator-controller-manager-547c58d5d9-pfr5x" Jul 23 17:50:02.915459 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.915442 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/740d1253-7e70-4b58-9c93-10cbd14d29d1-webhook-cert\") pod \"opendatahub-operator-controller-manager-547c58d5d9-pfr5x\" (UID: \"740d1253-7e70-4b58-9c93-10cbd14d29d1\") " pod="opendatahub/opendatahub-operator-controller-manager-547c58d5d9-pfr5x" Jul 23 17:50:02.924957 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.924888 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54c7q\" (UniqueName: \"kubernetes.io/projected/740d1253-7e70-4b58-9c93-10cbd14d29d1-kube-api-access-54c7q\") pod \"opendatahub-operator-controller-manager-547c58d5d9-pfr5x\" (UID: \"740d1253-7e70-4b58-9c93-10cbd14d29d1\") " pod="opendatahub/opendatahub-operator-controller-manager-547c58d5d9-pfr5x" Jul 23 17:50:02.981372 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:02.981336 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-547c58d5d9-pfr5x" Jul 23 17:50:03.122411 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:03.122374 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-547c58d5d9-pfr5x"] Jul 23 17:50:03.126440 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:50:03.126416 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740d1253_7e70_4b58_9c93_10cbd14d29d1.slice/crio-d4f275ea5d19dd9845ff1691367ff4fd1f7d2f4d16600077d09d62f0107a27f9 WatchSource:0}: Error finding container d4f275ea5d19dd9845ff1691367ff4fd1f7d2f4d16600077d09d62f0107a27f9: Status 404 returned error can't find the container with id d4f275ea5d19dd9845ff1691367ff4fd1f7d2f4d16600077d09d62f0107a27f9 Jul 23 17:50:03.128214 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:03.128191 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 17:50:03.943893 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:03.943855 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-547c58d5d9-pfr5x" event={"ID":"740d1253-7e70-4b58-9c93-10cbd14d29d1","Type":"ContainerStarted","Data":"d4f275ea5d19dd9845ff1691367ff4fd1f7d2f4d16600077d09d62f0107a27f9"} Jul 23 17:50:06.959434 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:06.959389 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-547c58d5d9-pfr5x" event={"ID":"740d1253-7e70-4b58-9c93-10cbd14d29d1","Type":"ContainerStarted","Data":"163838bc53db5edba6935ed70c11a7a13f29dc4295bb2952e8443867610685b5"} Jul 23 17:50:07.965563 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:07.965526 2565 generic.go:356] "Generic (PLEG): container finished" podID="740d1253-7e70-4b58-9c93-10cbd14d29d1" containerID="163838bc53db5edba6935ed70c11a7a13f29dc4295bb2952e8443867610685b5" exitCode=0 Jul 23 17:50:07.966055 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:07.965633 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-547c58d5d9-pfr5x" event={"ID":"740d1253-7e70-4b58-9c93-10cbd14d29d1","Type":"ContainerDied","Data":"163838bc53db5edba6935ed70c11a7a13f29dc4295bb2952e8443867610685b5"} Jul 23 17:50:08.972186 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:08.972146 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-547c58d5d9-pfr5x" event={"ID":"740d1253-7e70-4b58-9c93-10cbd14d29d1","Type":"ContainerStarted","Data":"ae4a6f6e5a64b8ba61ac92266b1d0f5269cee1300aee291ab99fceafc1d61167"} Jul 23 17:50:08.972664 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:08.972268 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-547c58d5d9-pfr5x" Jul 23 17:50:09.002405 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:09.002360 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-547c58d5d9-pfr5x" podStartSLOduration=2.5454085109999998 podStartE2EDuration="7.002343971s" totalImagesPullingTime="3.441844198s" totalInitContainerRuntime="1.015091262s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:50:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:50:08.999925973 +0000 UTC m=+353.078229336" watchObservedRunningTime="2026-07-23 17:50:09.002343971 +0000 UTC m=+353.080647383" Jul 23 17:50:19.979019 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:19.978987 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-547c58d5d9-pfr5x" Jul 23 17:50:32.591651 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:32.591612 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-754bfdc447-jwg9h"] Jul 23 17:50:32.605265 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:32.605241 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-754bfdc447-jwg9h" Jul 23 17:50:32.609244 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:32.609218 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-754bfdc447-jwg9h"] Jul 23 17:50:32.609502 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:32.609487 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 23 17:50:32.610250 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:32.610229 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 23 17:50:32.610357 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:32.610278 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 23 17:50:32.610357 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:32.610305 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-r55r8\"" Jul 23 17:50:32.610556 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:32.610539 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 23 17:50:32.610556 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:32.610551 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 23 17:50:32.763143 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:32.763108 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s92wp\" (UniqueName: \"kubernetes.io/projected/5691d164-7cef-4621-a094-4f0879d05407-kube-api-access-s92wp\") pod \"lws-controller-manager-754bfdc447-jwg9h\" (UID: \"5691d164-7cef-4621-a094-4f0879d05407\") " pod="openshift-lws-operator/lws-controller-manager-754bfdc447-jwg9h" Jul 23 17:50:32.763324 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:32.763185 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5691d164-7cef-4621-a094-4f0879d05407-manager-config\") pod \"lws-controller-manager-754bfdc447-jwg9h\" (UID: \"5691d164-7cef-4621-a094-4f0879d05407\") " pod="openshift-lws-operator/lws-controller-manager-754bfdc447-jwg9h" Jul 23 17:50:32.763324 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:32.763216 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5691d164-7cef-4621-a094-4f0879d05407-metrics-cert\") pod \"lws-controller-manager-754bfdc447-jwg9h\" (UID: \"5691d164-7cef-4621-a094-4f0879d05407\") " pod="openshift-lws-operator/lws-controller-manager-754bfdc447-jwg9h" Jul 23 17:50:32.763324 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:32.763241 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5691d164-7cef-4621-a094-4f0879d05407-cert\") pod \"lws-controller-manager-754bfdc447-jwg9h\" (UID: \"5691d164-7cef-4621-a094-4f0879d05407\") " pod="openshift-lws-operator/lws-controller-manager-754bfdc447-jwg9h" Jul 23 17:50:32.864365 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:32.864277 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5691d164-7cef-4621-a094-4f0879d05407-manager-config\") pod \"lws-controller-manager-754bfdc447-jwg9h\" (UID: \"5691d164-7cef-4621-a094-4f0879d05407\") " pod="openshift-lws-operator/lws-controller-manager-754bfdc447-jwg9h" Jul 23 17:50:32.864365 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:32.864312 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5691d164-7cef-4621-a094-4f0879d05407-metrics-cert\") pod \"lws-controller-manager-754bfdc447-jwg9h\" (UID: \"5691d164-7cef-4621-a094-4f0879d05407\") " pod="openshift-lws-operator/lws-controller-manager-754bfdc447-jwg9h" Jul 23 17:50:32.864365 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:32.864335 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5691d164-7cef-4621-a094-4f0879d05407-cert\") pod \"lws-controller-manager-754bfdc447-jwg9h\" (UID: \"5691d164-7cef-4621-a094-4f0879d05407\") " pod="openshift-lws-operator/lws-controller-manager-754bfdc447-jwg9h" Jul 23 17:50:32.864365 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:32.864371 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s92wp\" (UniqueName: \"kubernetes.io/projected/5691d164-7cef-4621-a094-4f0879d05407-kube-api-access-s92wp\") pod \"lws-controller-manager-754bfdc447-jwg9h\" (UID: \"5691d164-7cef-4621-a094-4f0879d05407\") " pod="openshift-lws-operator/lws-controller-manager-754bfdc447-jwg9h" Jul 23 17:50:32.865104 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:32.865083 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5691d164-7cef-4621-a094-4f0879d05407-manager-config\") pod \"lws-controller-manager-754bfdc447-jwg9h\" (UID: \"5691d164-7cef-4621-a094-4f0879d05407\") " pod="openshift-lws-operator/lws-controller-manager-754bfdc447-jwg9h" Jul 23 17:50:32.866910 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:32.866891 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5691d164-7cef-4621-a094-4f0879d05407-cert\") pod \"lws-controller-manager-754bfdc447-jwg9h\" (UID: \"5691d164-7cef-4621-a094-4f0879d05407\") " pod="openshift-lws-operator/lws-controller-manager-754bfdc447-jwg9h" Jul 23 17:50:32.866979 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:32.866952 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5691d164-7cef-4621-a094-4f0879d05407-metrics-cert\") pod \"lws-controller-manager-754bfdc447-jwg9h\" (UID: \"5691d164-7cef-4621-a094-4f0879d05407\") " pod="openshift-lws-operator/lws-controller-manager-754bfdc447-jwg9h" Jul 23 17:50:32.889669 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:32.889638 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s92wp\" (UniqueName: \"kubernetes.io/projected/5691d164-7cef-4621-a094-4f0879d05407-kube-api-access-s92wp\") pod \"lws-controller-manager-754bfdc447-jwg9h\" (UID: \"5691d164-7cef-4621-a094-4f0879d05407\") " pod="openshift-lws-operator/lws-controller-manager-754bfdc447-jwg9h" Jul 23 17:50:32.915612 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:32.915575 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-754bfdc447-jwg9h" Jul 23 17:50:33.067362 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:33.067324 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-754bfdc447-jwg9h"] Jul 23 17:50:33.070714 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:50:33.070688 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5691d164_7cef_4621_a094_4f0879d05407.slice/crio-a98ed5e9ca91adead4ee39b715fa9a52d9abf319e10c052491eb36304798a3ee WatchSource:0}: Error finding container a98ed5e9ca91adead4ee39b715fa9a52d9abf319e10c052491eb36304798a3ee: Status 404 returned error can't find the container with id a98ed5e9ca91adead4ee39b715fa9a52d9abf319e10c052491eb36304798a3ee Jul 23 17:50:33.080716 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:33.080692 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-754bfdc447-jwg9h" event={"ID":"5691d164-7cef-4621-a094-4f0879d05407","Type":"ContainerStarted","Data":"a98ed5e9ca91adead4ee39b715fa9a52d9abf319e10c052491eb36304798a3ee"} Jul 23 17:50:35.063717 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:35.063098 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-sp7bq"] Jul 23 17:50:35.072088 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:35.072038 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-sp7bq" Jul 23 17:50:35.075228 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:35.074903 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 23 17:50:35.075228 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:35.074922 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-kjz9t\"" Jul 23 17:50:35.075861 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:35.075831 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-6c4zk"] Jul 23 17:50:35.080190 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:35.080169 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6c4zk" Jul 23 17:50:35.080534 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:35.080295 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-sp7bq"] Jul 23 17:50:35.084500 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:35.084098 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-nwhsd\"" Jul 23 17:50:35.084500 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:35.084355 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 23 17:50:35.093721 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:35.093586 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-6c4zk"] Jul 23 17:50:35.184424 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:35.184374 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6ntn\" (UniqueName: \"kubernetes.io/projected/fa68142b-8dd3-4e3f-aa2f-23a6598568e1-kube-api-access-g6ntn\") pod \"kserve-controller-manager-d6f4b65b8-sp7bq\" (UID: \"fa68142b-8dd3-4e3f-aa2f-23a6598568e1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-sp7bq" Jul 23 17:50:35.184424 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:35.184422 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvzs5\" (UniqueName: \"kubernetes.io/projected/d0c8c911-4f29-4cc0-b055-514528b1afd3-kube-api-access-dvzs5\") pod \"llmisvc-controller-manager-777c6bd7d5-6c4zk\" (UID: \"d0c8c911-4f29-4cc0-b055-514528b1afd3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6c4zk" Jul 23 17:50:35.184700 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:35.184558 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa68142b-8dd3-4e3f-aa2f-23a6598568e1-cert\") pod \"kserve-controller-manager-d6f4b65b8-sp7bq\" (UID: \"fa68142b-8dd3-4e3f-aa2f-23a6598568e1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-sp7bq" Jul 23 17:50:35.184700 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:35.184628 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0c8c911-4f29-4cc0-b055-514528b1afd3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6c4zk\" (UID: \"d0c8c911-4f29-4cc0-b055-514528b1afd3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6c4zk" Jul 23 17:50:35.285909 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:35.285855 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6ntn\" (UniqueName: \"kubernetes.io/projected/fa68142b-8dd3-4e3f-aa2f-23a6598568e1-kube-api-access-g6ntn\") pod \"kserve-controller-manager-d6f4b65b8-sp7bq\" (UID: \"fa68142b-8dd3-4e3f-aa2f-23a6598568e1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-sp7bq" Jul 23 17:50:35.285909 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:35.285906 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvzs5\" (UniqueName: \"kubernetes.io/projected/d0c8c911-4f29-4cc0-b055-514528b1afd3-kube-api-access-dvzs5\") pod \"llmisvc-controller-manager-777c6bd7d5-6c4zk\" (UID: \"d0c8c911-4f29-4cc0-b055-514528b1afd3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6c4zk" Jul 23 17:50:35.286149 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:35.285970 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa68142b-8dd3-4e3f-aa2f-23a6598568e1-cert\") pod \"kserve-controller-manager-d6f4b65b8-sp7bq\" (UID: \"fa68142b-8dd3-4e3f-aa2f-23a6598568e1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-sp7bq" Jul 23 17:50:35.286149 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:35.286002 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0c8c911-4f29-4cc0-b055-514528b1afd3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6c4zk\" (UID: \"d0c8c911-4f29-4cc0-b055-514528b1afd3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6c4zk" Jul 23 17:50:35.286149 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:50:35.286148 2565 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 23 17:50:35.286288 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:50:35.286221 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0c8c911-4f29-4cc0-b055-514528b1afd3-cert podName:d0c8c911-4f29-4cc0-b055-514528b1afd3 nodeName:}" failed. No retries permitted until 2026-07-23 17:50:35.786200623 +0000 UTC m=+379.864503958 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d0c8c911-4f29-4cc0-b055-514528b1afd3-cert") pod "llmisvc-controller-manager-777c6bd7d5-6c4zk" (UID: "d0c8c911-4f29-4cc0-b055-514528b1afd3") : secret "llmisvc-webhook-server-cert" not found Jul 23 17:50:35.286393 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:50:35.286368 2565 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 23 17:50:35.286502 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:50:35.286430 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa68142b-8dd3-4e3f-aa2f-23a6598568e1-cert podName:fa68142b-8dd3-4e3f-aa2f-23a6598568e1 nodeName:}" failed. No retries permitted until 2026-07-23 17:50:35.786413338 +0000 UTC m=+379.864716669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fa68142b-8dd3-4e3f-aa2f-23a6598568e1-cert") pod "kserve-controller-manager-d6f4b65b8-sp7bq" (UID: "fa68142b-8dd3-4e3f-aa2f-23a6598568e1") : secret "kserve-webhook-server-cert" not found Jul 23 17:50:35.354409 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:35.354327 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvzs5\" (UniqueName: \"kubernetes.io/projected/d0c8c911-4f29-4cc0-b055-514528b1afd3-kube-api-access-dvzs5\") pod \"llmisvc-controller-manager-777c6bd7d5-6c4zk\" (UID: \"d0c8c911-4f29-4cc0-b055-514528b1afd3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6c4zk" Jul 23 17:50:35.386408 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:35.386369 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6ntn\" (UniqueName: \"kubernetes.io/projected/fa68142b-8dd3-4e3f-aa2f-23a6598568e1-kube-api-access-g6ntn\") pod \"kserve-controller-manager-d6f4b65b8-sp7bq\" (UID: \"fa68142b-8dd3-4e3f-aa2f-23a6598568e1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-sp7bq" Jul 23 17:50:35.790939 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:35.790905 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa68142b-8dd3-4e3f-aa2f-23a6598568e1-cert\") pod \"kserve-controller-manager-d6f4b65b8-sp7bq\" (UID: \"fa68142b-8dd3-4e3f-aa2f-23a6598568e1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-sp7bq" Jul 23 17:50:35.790939 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:35.790945 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0c8c911-4f29-4cc0-b055-514528b1afd3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6c4zk\" (UID: \"d0c8c911-4f29-4cc0-b055-514528b1afd3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6c4zk" Jul 23 17:50:35.791276 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:50:35.791066 2565 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 23 17:50:35.791276 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:50:35.791127 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0c8c911-4f29-4cc0-b055-514528b1afd3-cert podName:d0c8c911-4f29-4cc0-b055-514528b1afd3 nodeName:}" failed. No retries permitted until 2026-07-23 17:50:36.791112511 +0000 UTC m=+380.869415842 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d0c8c911-4f29-4cc0-b055-514528b1afd3-cert") pod "llmisvc-controller-manager-777c6bd7d5-6c4zk" (UID: "d0c8c911-4f29-4cc0-b055-514528b1afd3") : secret "llmisvc-webhook-server-cert" not found Jul 23 17:50:35.793315 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:35.793289 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa68142b-8dd3-4e3f-aa2f-23a6598568e1-cert\") pod \"kserve-controller-manager-d6f4b65b8-sp7bq\" (UID: \"fa68142b-8dd3-4e3f-aa2f-23a6598568e1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-sp7bq" Jul 23 17:50:35.989381 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:35.989348 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-sp7bq" Jul 23 17:50:36.104398 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:36.104366 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-754bfdc447-jwg9h" event={"ID":"5691d164-7cef-4621-a094-4f0879d05407","Type":"ContainerStarted","Data":"5fa55bae41b58dbf04c3c2cb0a16b4762b2fa5b08fd383418e6037e9d0834db4"} Jul 23 17:50:36.104875 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:36.104437 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-754bfdc447-jwg9h" Jul 23 17:50:36.137282 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:36.137255 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-sp7bq"] Jul 23 17:50:36.139477 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:50:36.139449 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa68142b_8dd3_4e3f_aa2f_23a6598568e1.slice/crio-55a2bcad28214c15385d9178d8859d663e26a4d8b62e7ab7450cbed286fa097e WatchSource:0}: Error finding container 55a2bcad28214c15385d9178d8859d663e26a4d8b62e7ab7450cbed286fa097e: Status 404 returned error can't find the container with id 55a2bcad28214c15385d9178d8859d663e26a4d8b62e7ab7450cbed286fa097e Jul 23 17:50:36.148369 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:36.148322 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-754bfdc447-jwg9h" podStartSLOduration=1.362670299 podStartE2EDuration="4.148306912s" totalImagesPullingTime="2.785636613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:50:36.147794335 +0000 UTC m=+380.226097689" watchObservedRunningTime="2026-07-23 17:50:36.148306912 +0000 UTC m=+380.226610264" Jul 23 17:50:36.800142 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:36.800105 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0c8c911-4f29-4cc0-b055-514528b1afd3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6c4zk\" (UID: \"d0c8c911-4f29-4cc0-b055-514528b1afd3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6c4zk" Jul 23 17:50:36.802474 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:36.802449 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0c8c911-4f29-4cc0-b055-514528b1afd3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6c4zk\" (UID: \"d0c8c911-4f29-4cc0-b055-514528b1afd3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6c4zk" Jul 23 17:50:36.902440 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:36.902404 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6c4zk" Jul 23 17:50:37.063934 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:37.063903 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-6c4zk"] Jul 23 17:50:37.066365 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:50:37.066310 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd0c8c911_4f29_4cc0_b055_514528b1afd3.slice/crio-09361165a24a6833da354f3bba28392806f959d4f08125db2ac6245f329a4b9f WatchSource:0}: Error finding container 09361165a24a6833da354f3bba28392806f959d4f08125db2ac6245f329a4b9f: Status 404 returned error can't find the container with id 09361165a24a6833da354f3bba28392806f959d4f08125db2ac6245f329a4b9f Jul 23 17:50:37.111231 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:37.111186 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6c4zk" event={"ID":"d0c8c911-4f29-4cc0-b055-514528b1afd3","Type":"ContainerStarted","Data":"09361165a24a6833da354f3bba28392806f959d4f08125db2ac6245f329a4b9f"} Jul 23 17:50:37.112431 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:37.112399 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-sp7bq" event={"ID":"fa68142b-8dd3-4e3f-aa2f-23a6598568e1","Type":"ContainerStarted","Data":"55a2bcad28214c15385d9178d8859d663e26a4d8b62e7ab7450cbed286fa097e"} Jul 23 17:50:40.129113 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:40.129075 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-sp7bq" event={"ID":"fa68142b-8dd3-4e3f-aa2f-23a6598568e1","Type":"ContainerStarted","Data":"9c839e80deb0892461c5d2694b08d9e859acdf71352d53e53cafc424a131e282"} Jul 23 17:50:40.129524 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:40.129168 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-sp7bq" Jul 23 17:50:40.150404 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:40.150341 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-sp7bq" podStartSLOduration=2.097449708 podStartE2EDuration="5.150323831s" totalImagesPullingTime="3.052874123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:50:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:50:40.149106238 +0000 UTC m=+384.227409611" watchObservedRunningTime="2026-07-23 17:50:40.150323831 +0000 UTC m=+384.228627185" Jul 23 17:50:41.135606 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:41.135555 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6c4zk" event={"ID":"d0c8c911-4f29-4cc0-b055-514528b1afd3","Type":"ContainerStarted","Data":"12728e6a1995138dee4569dab6793c1b3cdaab95311450d90c5613e173f3e873"} Jul 23 17:50:41.153443 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:41.153398 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6c4zk" podStartSLOduration=2.903658604 podStartE2EDuration="6.153383155s" totalImagesPullingTime="3.249724551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:50:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:50:41.153261749 +0000 UTC m=+385.231565099" watchObservedRunningTime="2026-07-23 17:50:41.153383155 +0000 UTC m=+385.231686509" Jul 23 17:50:42.139559 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:42.139529 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6c4zk" Jul 23 17:50:47.115277 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:50:47.115244 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-754bfdc447-jwg9h" Jul 23 17:51:11.141435 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:11.141402 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-sp7bq" Jul 23 17:51:13.146746 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:13.146709 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6c4zk" Jul 23 17:51:40.321408 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.321327 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b4ff9cd5d-9dgmq"] Jul 23 17:51:40.323719 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.323696 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b4ff9cd5d-9dgmq" Jul 23 17:51:40.331523 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.331503 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-78r8z\"" Jul 23 17:51:40.332449 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.332428 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 23 17:51:40.332622 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.332440 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 23 17:51:40.332622 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.332444 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 23 17:51:40.332782 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.332466 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 23 17:51:40.332850 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.332833 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 23 17:51:40.341314 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.341292 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b4ff9cd5d-9dgmq"] Jul 23 17:51:40.353053 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.353023 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 23 17:51:40.514852 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.514814 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a2f67767-4863-4a8a-9999-99875fd52c19-service-ca\") pod \"console-6b4ff9cd5d-9dgmq\" (UID: \"a2f67767-4863-4a8a-9999-99875fd52c19\") " pod="openshift-console/console-6b4ff9cd5d-9dgmq" Jul 23 17:51:40.515026 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.514900 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a2f67767-4863-4a8a-9999-99875fd52c19-console-config\") pod \"console-6b4ff9cd5d-9dgmq\" (UID: \"a2f67767-4863-4a8a-9999-99875fd52c19\") " pod="openshift-console/console-6b4ff9cd5d-9dgmq" Jul 23 17:51:40.515026 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.514931 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvkn9\" (UniqueName: \"kubernetes.io/projected/a2f67767-4863-4a8a-9999-99875fd52c19-kube-api-access-gvkn9\") pod \"console-6b4ff9cd5d-9dgmq\" (UID: \"a2f67767-4863-4a8a-9999-99875fd52c19\") " pod="openshift-console/console-6b4ff9cd5d-9dgmq" Jul 23 17:51:40.515026 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.514970 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2f67767-4863-4a8a-9999-99875fd52c19-console-serving-cert\") pod \"console-6b4ff9cd5d-9dgmq\" (UID: \"a2f67767-4863-4a8a-9999-99875fd52c19\") " pod="openshift-console/console-6b4ff9cd5d-9dgmq" Jul 23 17:51:40.515026 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.515010 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a2f67767-4863-4a8a-9999-99875fd52c19-console-oauth-config\") pod \"console-6b4ff9cd5d-9dgmq\" (UID: \"a2f67767-4863-4a8a-9999-99875fd52c19\") " pod="openshift-console/console-6b4ff9cd5d-9dgmq" Jul 23 17:51:40.515186 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.515103 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a2f67767-4863-4a8a-9999-99875fd52c19-oauth-serving-cert\") pod \"console-6b4ff9cd5d-9dgmq\" (UID: \"a2f67767-4863-4a8a-9999-99875fd52c19\") " pod="openshift-console/console-6b4ff9cd5d-9dgmq" Jul 23 17:51:40.515186 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.515154 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2f67767-4863-4a8a-9999-99875fd52c19-trusted-ca-bundle\") pod \"console-6b4ff9cd5d-9dgmq\" (UID: \"a2f67767-4863-4a8a-9999-99875fd52c19\") " pod="openshift-console/console-6b4ff9cd5d-9dgmq" Jul 23 17:51:40.615994 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.615897 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a2f67767-4863-4a8a-9999-99875fd52c19-service-ca\") pod \"console-6b4ff9cd5d-9dgmq\" (UID: \"a2f67767-4863-4a8a-9999-99875fd52c19\") " pod="openshift-console/console-6b4ff9cd5d-9dgmq" Jul 23 17:51:40.615994 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.615945 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a2f67767-4863-4a8a-9999-99875fd52c19-console-config\") pod \"console-6b4ff9cd5d-9dgmq\" (UID: \"a2f67767-4863-4a8a-9999-99875fd52c19\") " pod="openshift-console/console-6b4ff9cd5d-9dgmq" Jul 23 17:51:40.615994 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.615962 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvkn9\" (UniqueName: \"kubernetes.io/projected/a2f67767-4863-4a8a-9999-99875fd52c19-kube-api-access-gvkn9\") pod \"console-6b4ff9cd5d-9dgmq\" (UID: \"a2f67767-4863-4a8a-9999-99875fd52c19\") " pod="openshift-console/console-6b4ff9cd5d-9dgmq" Jul 23 17:51:40.615994 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.615991 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2f67767-4863-4a8a-9999-99875fd52c19-console-serving-cert\") pod \"console-6b4ff9cd5d-9dgmq\" (UID: \"a2f67767-4863-4a8a-9999-99875fd52c19\") " pod="openshift-console/console-6b4ff9cd5d-9dgmq" Jul 23 17:51:40.616303 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.616024 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a2f67767-4863-4a8a-9999-99875fd52c19-console-oauth-config\") pod \"console-6b4ff9cd5d-9dgmq\" (UID: \"a2f67767-4863-4a8a-9999-99875fd52c19\") " pod="openshift-console/console-6b4ff9cd5d-9dgmq" Jul 23 17:51:40.616303 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.616064 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a2f67767-4863-4a8a-9999-99875fd52c19-oauth-serving-cert\") pod \"console-6b4ff9cd5d-9dgmq\" (UID: \"a2f67767-4863-4a8a-9999-99875fd52c19\") " pod="openshift-console/console-6b4ff9cd5d-9dgmq" Jul 23 17:51:40.616303 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.616187 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2f67767-4863-4a8a-9999-99875fd52c19-trusted-ca-bundle\") pod \"console-6b4ff9cd5d-9dgmq\" (UID: \"a2f67767-4863-4a8a-9999-99875fd52c19\") " pod="openshift-console/console-6b4ff9cd5d-9dgmq" Jul 23 17:51:40.616847 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.616821 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a2f67767-4863-4a8a-9999-99875fd52c19-oauth-serving-cert\") pod \"console-6b4ff9cd5d-9dgmq\" (UID: \"a2f67767-4863-4a8a-9999-99875fd52c19\") " pod="openshift-console/console-6b4ff9cd5d-9dgmq" Jul 23 17:51:40.616954 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.616845 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a2f67767-4863-4a8a-9999-99875fd52c19-console-config\") pod \"console-6b4ff9cd5d-9dgmq\" (UID: \"a2f67767-4863-4a8a-9999-99875fd52c19\") " pod="openshift-console/console-6b4ff9cd5d-9dgmq" Jul 23 17:51:40.616954 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.616822 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a2f67767-4863-4a8a-9999-99875fd52c19-service-ca\") pod \"console-6b4ff9cd5d-9dgmq\" (UID: \"a2f67767-4863-4a8a-9999-99875fd52c19\") " pod="openshift-console/console-6b4ff9cd5d-9dgmq" Jul 23 17:51:40.617034 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.617016 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2f67767-4863-4a8a-9999-99875fd52c19-trusted-ca-bundle\") pod \"console-6b4ff9cd5d-9dgmq\" (UID: \"a2f67767-4863-4a8a-9999-99875fd52c19\") " pod="openshift-console/console-6b4ff9cd5d-9dgmq" Jul 23 17:51:40.618699 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.618679 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a2f67767-4863-4a8a-9999-99875fd52c19-console-oauth-config\") pod \"console-6b4ff9cd5d-9dgmq\" (UID: \"a2f67767-4863-4a8a-9999-99875fd52c19\") " pod="openshift-console/console-6b4ff9cd5d-9dgmq" Jul 23 17:51:40.618843 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.618821 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2f67767-4863-4a8a-9999-99875fd52c19-console-serving-cert\") pod \"console-6b4ff9cd5d-9dgmq\" (UID: \"a2f67767-4863-4a8a-9999-99875fd52c19\") " pod="openshift-console/console-6b4ff9cd5d-9dgmq" Jul 23 17:51:40.625482 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.625460 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvkn9\" (UniqueName: \"kubernetes.io/projected/a2f67767-4863-4a8a-9999-99875fd52c19-kube-api-access-gvkn9\") pod \"console-6b4ff9cd5d-9dgmq\" (UID: \"a2f67767-4863-4a8a-9999-99875fd52c19\") " pod="openshift-console/console-6b4ff9cd5d-9dgmq" Jul 23 17:51:40.633185 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.633164 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b4ff9cd5d-9dgmq" Jul 23 17:51:40.769655 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:40.769590 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b4ff9cd5d-9dgmq"] Jul 23 17:51:40.771878 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:51:40.771851 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f67767_4863_4a8a_9999_99875fd52c19.slice/crio-b88572445ee198dc3ed171e6ae72014bb0f727b2d8beea6bcfd7a552108c5972 WatchSource:0}: Error finding container b88572445ee198dc3ed171e6ae72014bb0f727b2d8beea6bcfd7a552108c5972: Status 404 returned error can't find the container with id b88572445ee198dc3ed171e6ae72014bb0f727b2d8beea6bcfd7a552108c5972 Jul 23 17:51:41.402146 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:41.402103 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b4ff9cd5d-9dgmq" event={"ID":"a2f67767-4863-4a8a-9999-99875fd52c19","Type":"ContainerStarted","Data":"7123d9bb74b0c17f2e0325b9993c12cbddc6c9927dc557c1b3f4b4f50213205c"} Jul 23 17:51:41.402660 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:41.402193 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b4ff9cd5d-9dgmq" event={"ID":"a2f67767-4863-4a8a-9999-99875fd52c19","Type":"ContainerStarted","Data":"b88572445ee198dc3ed171e6ae72014bb0f727b2d8beea6bcfd7a552108c5972"} Jul 23 17:51:41.430474 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:41.430420 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b4ff9cd5d-9dgmq" podStartSLOduration=1.430406566 podStartE2EDuration="1.430406566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:51:41.42913168 +0000 UTC m=+445.507435032" watchObservedRunningTime="2026-07-23 17:51:41.430406566 +0000 UTC m=+445.508709918" Jul 23 17:51:50.633865 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:50.633819 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b4ff9cd5d-9dgmq" Jul 23 17:51:50.634329 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:50.633903 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b4ff9cd5d-9dgmq" Jul 23 17:51:50.639013 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:50.638988 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b4ff9cd5d-9dgmq" Jul 23 17:51:51.457909 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:51:51.457878 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b4ff9cd5d-9dgmq" Jul 23 17:52:24.145008 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:52:24.144970 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2z86"] Jul 23 17:52:24.148477 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:52:24.148460 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2z86" Jul 23 17:52:24.156230 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:52:24.156209 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 17:52:24.156355 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:52:24.156250 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 17:52:24.156355 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:52:24.156339 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jpgwl\"" Jul 23 17:52:24.182736 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:52:24.182700 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2z86"] Jul 23 17:52:24.276459 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:52:24.276428 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttznl\" (UniqueName: \"kubernetes.io/projected/1c6fdb49-b699-4fe8-9834-6a335376069d-kube-api-access-ttznl\") pod \"kuadrant-operator-controller-manager-86878c769f-b2z86\" (UID: \"1c6fdb49-b699-4fe8-9834-6a335376069d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2z86" Jul 23 17:52:24.276662 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:52:24.276476 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c6fdb49-b699-4fe8-9834-6a335376069d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-b2z86\" (UID: \"1c6fdb49-b699-4fe8-9834-6a335376069d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2z86" Jul 23 17:52:24.377298 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:52:24.377246 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttznl\" (UniqueName: \"kubernetes.io/projected/1c6fdb49-b699-4fe8-9834-6a335376069d-kube-api-access-ttznl\") pod \"kuadrant-operator-controller-manager-86878c769f-b2z86\" (UID: \"1c6fdb49-b699-4fe8-9834-6a335376069d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2z86" Jul 23 17:52:24.377472 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:52:24.377314 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c6fdb49-b699-4fe8-9834-6a335376069d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-b2z86\" (UID: \"1c6fdb49-b699-4fe8-9834-6a335376069d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2z86" Jul 23 17:52:24.377683 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:52:24.377663 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c6fdb49-b699-4fe8-9834-6a335376069d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-b2z86\" (UID: \"1c6fdb49-b699-4fe8-9834-6a335376069d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2z86" Jul 23 17:52:24.402903 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:52:24.402836 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttznl\" (UniqueName: \"kubernetes.io/projected/1c6fdb49-b699-4fe8-9834-6a335376069d-kube-api-access-ttznl\") pod \"kuadrant-operator-controller-manager-86878c769f-b2z86\" (UID: \"1c6fdb49-b699-4fe8-9834-6a335376069d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2z86" Jul 23 17:52:24.457959 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:52:24.457923 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2z86" Jul 23 17:52:24.629093 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:52:24.629063 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2z86"] Jul 23 17:52:24.629768 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:52:24.629734 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c6fdb49_b699_4fe8_9834_6a335376069d.slice/crio-5cd90887032575f1097f0b5461d51e3ff89fbf105ae345a01bb61ada436cca81 WatchSource:0}: Error finding container 5cd90887032575f1097f0b5461d51e3ff89fbf105ae345a01bb61ada436cca81: Status 404 returned error can't find the container with id 5cd90887032575f1097f0b5461d51e3ff89fbf105ae345a01bb61ada436cca81 Jul 23 17:52:25.612106 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:52:25.612064 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2z86" event={"ID":"1c6fdb49-b699-4fe8-9834-6a335376069d","Type":"ContainerStarted","Data":"5cd90887032575f1097f0b5461d51e3ff89fbf105ae345a01bb61ada436cca81"} Jul 23 17:52:29.633509 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:52:29.633409 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2z86" event={"ID":"1c6fdb49-b699-4fe8-9834-6a335376069d","Type":"ContainerStarted","Data":"21c89547b5e24206ac56b311abd199031c64247568373dbbccf8d17a8f4a2062"} Jul 23 17:52:29.633988 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:52:29.633541 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2z86" Jul 23 17:52:29.693986 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:52:29.693939 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2z86" podStartSLOduration=0.971108149 podStartE2EDuration="5.693923415s" totalImagesPullingTime="4.722815266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:52:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:52:29.692554624 +0000 UTC m=+493.770857978" watchObservedRunningTime="2026-07-23 17:52:29.693923415 +0000 UTC m=+493.772226767" Jul 23 17:52:40.639839 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:52:40.639802 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2z86" Jul 23 17:54:16.333502 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:16.333470 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-m7k4j_1cca3236-5c90-4abb-9b9b-39e40b1903c3/console-operator/1.log" Jul 23 17:54:16.333973 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:16.333872 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-m7k4j_1cca3236-5c90-4abb-9b9b-39e40b1903c3/console-operator/1.log" Jul 23 17:54:17.555414 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:17.555379 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6755c5b8f8-xtm6x"] Jul 23 17:54:17.558462 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:17.558444 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6755c5b8f8-xtm6x" Jul 23 17:54:17.563460 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:17.563443 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 23 17:54:17.563612 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:17.563580 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 17:54:17.563785 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:17.563765 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zlv7p\"" Jul 23 17:54:17.564698 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:17.564680 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 23 17:54:17.573741 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:17.573715 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6755c5b8f8-xtm6x"] Jul 23 17:54:17.618853 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:17.618808 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca4b9476-9cf2-44f1-95b3-51215376a655-openshift-service-ca\") pod \"maas-api-6755c5b8f8-xtm6x\" (UID: \"ca4b9476-9cf2-44f1-95b3-51215376a655\") " pod="odh-ai-gateway-infra/maas-api-6755c5b8f8-xtm6x" Jul 23 17:54:17.619015 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:17.618867 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca4b9476-9cf2-44f1-95b3-51215376a655-maas-api-tls\") pod \"maas-api-6755c5b8f8-xtm6x\" (UID: \"ca4b9476-9cf2-44f1-95b3-51215376a655\") " pod="odh-ai-gateway-infra/maas-api-6755c5b8f8-xtm6x" Jul 23 17:54:17.619015 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:17.618994 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxggc\" (UniqueName: \"kubernetes.io/projected/ca4b9476-9cf2-44f1-95b3-51215376a655-kube-api-access-bxggc\") pod \"maas-api-6755c5b8f8-xtm6x\" (UID: \"ca4b9476-9cf2-44f1-95b3-51215376a655\") " pod="odh-ai-gateway-infra/maas-api-6755c5b8f8-xtm6x" Jul 23 17:54:17.719972 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:17.719918 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxggc\" (UniqueName: \"kubernetes.io/projected/ca4b9476-9cf2-44f1-95b3-51215376a655-kube-api-access-bxggc\") pod \"maas-api-6755c5b8f8-xtm6x\" (UID: \"ca4b9476-9cf2-44f1-95b3-51215376a655\") " pod="odh-ai-gateway-infra/maas-api-6755c5b8f8-xtm6x" Jul 23 17:54:17.720147 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:17.720000 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca4b9476-9cf2-44f1-95b3-51215376a655-openshift-service-ca\") pod \"maas-api-6755c5b8f8-xtm6x\" (UID: \"ca4b9476-9cf2-44f1-95b3-51215376a655\") " pod="odh-ai-gateway-infra/maas-api-6755c5b8f8-xtm6x" Jul 23 17:54:17.720147 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:17.720044 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca4b9476-9cf2-44f1-95b3-51215376a655-maas-api-tls\") pod \"maas-api-6755c5b8f8-xtm6x\" (UID: \"ca4b9476-9cf2-44f1-95b3-51215376a655\") " pod="odh-ai-gateway-infra/maas-api-6755c5b8f8-xtm6x" Jul 23 17:54:17.720245 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:54:17.720223 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 23 17:54:17.720326 ip-10-0-137-82 kubenswrapper[2565]: E0723 17:54:17.720314 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca4b9476-9cf2-44f1-95b3-51215376a655-maas-api-tls podName:ca4b9476-9cf2-44f1-95b3-51215376a655 nodeName:}" failed. No retries permitted until 2026-07-23 17:54:18.220292987 +0000 UTC m=+602.298596335 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ca4b9476-9cf2-44f1-95b3-51215376a655-maas-api-tls") pod "maas-api-6755c5b8f8-xtm6x" (UID: "ca4b9476-9cf2-44f1-95b3-51215376a655") : secret "maas-api-serving-cert" not found Jul 23 17:54:17.720686 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:17.720669 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca4b9476-9cf2-44f1-95b3-51215376a655-openshift-service-ca\") pod \"maas-api-6755c5b8f8-xtm6x\" (UID: \"ca4b9476-9cf2-44f1-95b3-51215376a655\") " pod="odh-ai-gateway-infra/maas-api-6755c5b8f8-xtm6x" Jul 23 17:54:17.729097 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:17.729070 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxggc\" (UniqueName: \"kubernetes.io/projected/ca4b9476-9cf2-44f1-95b3-51215376a655-kube-api-access-bxggc\") pod \"maas-api-6755c5b8f8-xtm6x\" (UID: \"ca4b9476-9cf2-44f1-95b3-51215376a655\") " pod="odh-ai-gateway-infra/maas-api-6755c5b8f8-xtm6x" Jul 23 17:54:18.145653 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:18.145611 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-kq59n"] Jul 23 17:54:18.148971 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:18.148951 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-kq59n" Jul 23 17:54:18.152127 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:18.152103 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-4qrmx\"" Jul 23 17:54:18.152318 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:18.152304 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 23 17:54:18.160051 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:18.160027 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-kq59n"] Jul 23 17:54:18.225182 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:18.225153 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d5a8e69-cc15-4912-952b-272908244643-plugins-config-volume\") pod \"payload-processing-5bf47cd988-kq59n\" (UID: \"3d5a8e69-cc15-4912-952b-272908244643\") " pod="openshift-ingress/payload-processing-5bf47cd988-kq59n" Jul 23 17:54:18.225368 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:18.225219 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca4b9476-9cf2-44f1-95b3-51215376a655-maas-api-tls\") pod \"maas-api-6755c5b8f8-xtm6x\" (UID: \"ca4b9476-9cf2-44f1-95b3-51215376a655\") " pod="odh-ai-gateway-infra/maas-api-6755c5b8f8-xtm6x" Jul 23 17:54:18.225368 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:18.225257 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btnsw\" (UniqueName: \"kubernetes.io/projected/3d5a8e69-cc15-4912-952b-272908244643-kube-api-access-btnsw\") pod \"payload-processing-5bf47cd988-kq59n\" (UID: \"3d5a8e69-cc15-4912-952b-272908244643\") " pod="openshift-ingress/payload-processing-5bf47cd988-kq59n" Jul 23 17:54:18.227682 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:18.227663 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca4b9476-9cf2-44f1-95b3-51215376a655-maas-api-tls\") pod \"maas-api-6755c5b8f8-xtm6x\" (UID: \"ca4b9476-9cf2-44f1-95b3-51215376a655\") " pod="odh-ai-gateway-infra/maas-api-6755c5b8f8-xtm6x" Jul 23 17:54:18.326313 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:18.326266 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btnsw\" (UniqueName: \"kubernetes.io/projected/3d5a8e69-cc15-4912-952b-272908244643-kube-api-access-btnsw\") pod \"payload-processing-5bf47cd988-kq59n\" (UID: \"3d5a8e69-cc15-4912-952b-272908244643\") " pod="openshift-ingress/payload-processing-5bf47cd988-kq59n" Jul 23 17:54:18.326472 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:18.326366 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d5a8e69-cc15-4912-952b-272908244643-plugins-config-volume\") pod \"payload-processing-5bf47cd988-kq59n\" (UID: \"3d5a8e69-cc15-4912-952b-272908244643\") " pod="openshift-ingress/payload-processing-5bf47cd988-kq59n" Jul 23 17:54:18.327107 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:18.327087 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d5a8e69-cc15-4912-952b-272908244643-plugins-config-volume\") pod \"payload-processing-5bf47cd988-kq59n\" (UID: \"3d5a8e69-cc15-4912-952b-272908244643\") " pod="openshift-ingress/payload-processing-5bf47cd988-kq59n" Jul 23 17:54:18.336826 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:18.336798 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btnsw\" (UniqueName: \"kubernetes.io/projected/3d5a8e69-cc15-4912-952b-272908244643-kube-api-access-btnsw\") pod \"payload-processing-5bf47cd988-kq59n\" (UID: \"3d5a8e69-cc15-4912-952b-272908244643\") " pod="openshift-ingress/payload-processing-5bf47cd988-kq59n" Jul 23 17:54:18.459984 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:18.459894 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-kq59n" Jul 23 17:54:18.469110 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:18.469084 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6755c5b8f8-xtm6x" Jul 23 17:54:18.605667 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:18.605550 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-kq59n"] Jul 23 17:54:18.607400 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:54:18.607373 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d5a8e69_cc15_4912_952b_272908244643.slice/crio-45e802748bc41db31a8e00f2ff9692b630784efeb5baa4fcf6bcf62a9910e66e WatchSource:0}: Error finding container 45e802748bc41db31a8e00f2ff9692b630784efeb5baa4fcf6bcf62a9910e66e: Status 404 returned error can't find the container with id 45e802748bc41db31a8e00f2ff9692b630784efeb5baa4fcf6bcf62a9910e66e Jul 23 17:54:18.638870 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:18.638841 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6755c5b8f8-xtm6x"] Jul 23 17:54:18.641689 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:54:18.641658 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca4b9476_9cf2_44f1_95b3_51215376a655.slice/crio-25d695596ff5654d27cdb7ace798cbded7f8cdb36c7e7beb7a1acf0e2e019b7f WatchSource:0}: Error finding container 25d695596ff5654d27cdb7ace798cbded7f8cdb36c7e7beb7a1acf0e2e019b7f: Status 404 returned error can't find the container with id 25d695596ff5654d27cdb7ace798cbded7f8cdb36c7e7beb7a1acf0e2e019b7f Jul 23 17:54:19.125306 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:19.125272 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-kq59n" event={"ID":"3d5a8e69-cc15-4912-952b-272908244643","Type":"ContainerStarted","Data":"45e802748bc41db31a8e00f2ff9692b630784efeb5baa4fcf6bcf62a9910e66e"} Jul 23 17:54:19.129159 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:19.129127 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6755c5b8f8-xtm6x" event={"ID":"ca4b9476-9cf2-44f1-95b3-51215376a655","Type":"ContainerStarted","Data":"25d695596ff5654d27cdb7ace798cbded7f8cdb36c7e7beb7a1acf0e2e019b7f"} Jul 23 17:54:23.151551 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:23.151514 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-kq59n" event={"ID":"3d5a8e69-cc15-4912-952b-272908244643","Type":"ContainerStarted","Data":"f0dac8832191f4abc0fd49222ac1b862f6784a126404b90028b6176340c26495"} Jul 23 17:54:23.152030 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:23.151584 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-kq59n" Jul 23 17:54:23.153022 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:23.152998 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6755c5b8f8-xtm6x" event={"ID":"ca4b9476-9cf2-44f1-95b3-51215376a655","Type":"ContainerStarted","Data":"95425d17c95a47eb3777b6d5a5a9ef323d893b438377cea05aa5b7036a0d1e9a"} Jul 23 17:54:23.153136 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:23.153041 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6755c5b8f8-xtm6x" Jul 23 17:54:23.170519 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:23.170474 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-kq59n" podStartSLOduration=1.678068071 podStartE2EDuration="5.170461776s" totalImagesPullingTime="3.492393705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:54:23.169769544 +0000 UTC m=+607.248072897" watchObservedRunningTime="2026-07-23 17:54:23.170461776 +0000 UTC m=+607.248765128" Jul 23 17:54:23.188020 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:23.187978 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6755c5b8f8-xtm6x" podStartSLOduration=2.771596244 podStartE2EDuration="6.187964199s" totalImagesPullingTime="3.416367955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:54:23.186887135 +0000 UTC m=+607.265190508" watchObservedRunningTime="2026-07-23 17:54:23.187964199 +0000 UTC m=+607.266267553" Jul 23 17:54:29.163413 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:29.163378 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6755c5b8f8-xtm6x" Jul 23 17:54:34.159956 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:54:34.159923 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-kq59n" Jul 23 17:55:07.989939 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:07.989901 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q"] Jul 23 17:55:07.992223 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:07.992203 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q" Jul 23 17:55:07.995098 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:07.995074 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9tdlq\"" Jul 23 17:55:07.995529 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:07.995505 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 23 17:55:07.995648 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:07.995515 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 23 17:55:07.995981 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:07.995961 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 23 17:55:08.007445 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:08.007418 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q"] Jul 23 17:55:08.058811 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:08.058774 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64bd615c-b608-423d-a4cd-d6122be3c236-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q\" (UID: \"64bd615c-b608-423d-a4cd-d6122be3c236\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q" Jul 23 17:55:08.058811 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:08.058808 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64bd615c-b608-423d-a4cd-d6122be3c236-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q\" (UID: \"64bd615c-b608-423d-a4cd-d6122be3c236\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q" Jul 23 17:55:08.059054 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:08.058876 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgpj6\" (UniqueName: \"kubernetes.io/projected/64bd615c-b608-423d-a4cd-d6122be3c236-kube-api-access-zgpj6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q\" (UID: \"64bd615c-b608-423d-a4cd-d6122be3c236\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q" Jul 23 17:55:08.059054 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:08.058906 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64bd615c-b608-423d-a4cd-d6122be3c236-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q\" (UID: \"64bd615c-b608-423d-a4cd-d6122be3c236\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q" Jul 23 17:55:08.059054 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:08.058934 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64bd615c-b608-423d-a4cd-d6122be3c236-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q\" (UID: \"64bd615c-b608-423d-a4cd-d6122be3c236\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q" Jul 23 17:55:08.059054 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:08.058967 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64bd615c-b608-423d-a4cd-d6122be3c236-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q\" (UID: \"64bd615c-b608-423d-a4cd-d6122be3c236\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q" Jul 23 17:55:08.160258 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:08.160211 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64bd615c-b608-423d-a4cd-d6122be3c236-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q\" (UID: \"64bd615c-b608-423d-a4cd-d6122be3c236\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q" Jul 23 17:55:08.160258 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:08.160270 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64bd615c-b608-423d-a4cd-d6122be3c236-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q\" (UID: \"64bd615c-b608-423d-a4cd-d6122be3c236\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q" Jul 23 17:55:08.160553 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:08.160291 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64bd615c-b608-423d-a4cd-d6122be3c236-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q\" (UID: \"64bd615c-b608-423d-a4cd-d6122be3c236\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q" Jul 23 17:55:08.160553 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:08.160335 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgpj6\" (UniqueName: \"kubernetes.io/projected/64bd615c-b608-423d-a4cd-d6122be3c236-kube-api-access-zgpj6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q\" (UID: \"64bd615c-b608-423d-a4cd-d6122be3c236\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q" Jul 23 17:55:08.160553 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:08.160362 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64bd615c-b608-423d-a4cd-d6122be3c236-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q\" (UID: \"64bd615c-b608-423d-a4cd-d6122be3c236\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q" Jul 23 17:55:08.160553 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:08.160513 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64bd615c-b608-423d-a4cd-d6122be3c236-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q\" (UID: \"64bd615c-b608-423d-a4cd-d6122be3c236\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q" Jul 23 17:55:08.160803 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:08.160734 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64bd615c-b608-423d-a4cd-d6122be3c236-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q\" (UID: \"64bd615c-b608-423d-a4cd-d6122be3c236\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q" Jul 23 17:55:08.160803 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:08.160770 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64bd615c-b608-423d-a4cd-d6122be3c236-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q\" (UID: \"64bd615c-b608-423d-a4cd-d6122be3c236\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q" Jul 23 17:55:08.160871 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:08.160861 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64bd615c-b608-423d-a4cd-d6122be3c236-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q\" (UID: \"64bd615c-b608-423d-a4cd-d6122be3c236\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q" Jul 23 17:55:08.162547 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:08.162526 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64bd615c-b608-423d-a4cd-d6122be3c236-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q\" (UID: \"64bd615c-b608-423d-a4cd-d6122be3c236\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q" Jul 23 17:55:08.162860 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:08.162845 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64bd615c-b608-423d-a4cd-d6122be3c236-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q\" (UID: \"64bd615c-b608-423d-a4cd-d6122be3c236\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q" Jul 23 17:55:08.169208 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:08.169181 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgpj6\" (UniqueName: \"kubernetes.io/projected/64bd615c-b608-423d-a4cd-d6122be3c236-kube-api-access-zgpj6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q\" (UID: \"64bd615c-b608-423d-a4cd-d6122be3c236\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q" Jul 23 17:55:08.302691 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:08.302651 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q" Jul 23 17:55:08.437854 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:08.437822 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q"] Jul 23 17:55:08.439671 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:55:08.439643 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64bd615c_b608_423d_a4cd_d6122be3c236.slice/crio-fdfb5ee5ecbcab8b6b6cba477deec67cbdc7c8aa188c6002707378ef1e003890 WatchSource:0}: Error finding container fdfb5ee5ecbcab8b6b6cba477deec67cbdc7c8aa188c6002707378ef1e003890: Status 404 returned error can't find the container with id fdfb5ee5ecbcab8b6b6cba477deec67cbdc7c8aa188c6002707378ef1e003890 Jul 23 17:55:08.441262 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:08.441241 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 17:55:09.376400 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:09.376358 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q" event={"ID":"64bd615c-b608-423d-a4cd-d6122be3c236","Type":"ContainerStarted","Data":"fdfb5ee5ecbcab8b6b6cba477deec67cbdc7c8aa188c6002707378ef1e003890"} Jul 23 17:55:10.382372 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:10.382341 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q" event={"ID":"64bd615c-b608-423d-a4cd-d6122be3c236","Type":"ContainerStarted","Data":"59f4a023fc2a14403b03f0d67930a1b68a77563ab3715b6b066853e43f0df5ae"} Jul 23 17:55:10.404206 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:10.404148 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q" podStartSLOduration=1.959565747 podStartE2EDuration="3.40413077s" totalImagesPullingTime="1.444565023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:55:10.40366864 +0000 UTC m=+654.481972015" watchObservedRunningTime="2026-07-23 17:55:10.40413077 +0000 UTC m=+654.482434124" Jul 23 17:55:18.303210 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:18.303158 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q" Jul 23 17:55:18.303677 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:18.303308 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q" Jul 23 17:55:18.315945 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:18.315921 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q" Jul 23 17:55:18.431418 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:18.431392 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q" Jul 23 17:55:38.698767 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:38.698734 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zm89g"] Jul 23 17:55:38.705488 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:38.705467 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm89g" Jul 23 17:55:38.708964 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:38.708941 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 23 17:55:38.717277 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:38.717250 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zm89g"] Jul 23 17:55:38.728298 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:38.728273 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7bb31f1-07ea-4418-b744-fb09ba8f3f53-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm89g\" (UID: \"d7bb31f1-07ea-4418-b744-fb09ba8f3f53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm89g" Jul 23 17:55:38.728420 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:38.728304 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7bb31f1-07ea-4418-b744-fb09ba8f3f53-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm89g\" (UID: \"d7bb31f1-07ea-4418-b744-fb09ba8f3f53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm89g" Jul 23 17:55:38.728420 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:38.728414 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7bb31f1-07ea-4418-b744-fb09ba8f3f53-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm89g\" (UID: \"d7bb31f1-07ea-4418-b744-fb09ba8f3f53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm89g" Jul 23 17:55:38.728521 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:38.728469 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7bb31f1-07ea-4418-b744-fb09ba8f3f53-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm89g\" (UID: \"d7bb31f1-07ea-4418-b744-fb09ba8f3f53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm89g" Jul 23 17:55:38.728521 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:38.728496 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6p82\" (UniqueName: \"kubernetes.io/projected/d7bb31f1-07ea-4418-b744-fb09ba8f3f53-kube-api-access-d6p82\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm89g\" (UID: \"d7bb31f1-07ea-4418-b744-fb09ba8f3f53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm89g" Jul 23 17:55:38.728644 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:38.728577 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7bb31f1-07ea-4418-b744-fb09ba8f3f53-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm89g\" (UID: \"d7bb31f1-07ea-4418-b744-fb09ba8f3f53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm89g" Jul 23 17:55:38.830075 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:38.830032 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7bb31f1-07ea-4418-b744-fb09ba8f3f53-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm89g\" (UID: \"d7bb31f1-07ea-4418-b744-fb09ba8f3f53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm89g" Jul 23 17:55:38.830265 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:38.830083 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7bb31f1-07ea-4418-b744-fb09ba8f3f53-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm89g\" (UID: \"d7bb31f1-07ea-4418-b744-fb09ba8f3f53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm89g" Jul 23 17:55:38.830265 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:38.830113 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6p82\" (UniqueName: \"kubernetes.io/projected/d7bb31f1-07ea-4418-b744-fb09ba8f3f53-kube-api-access-d6p82\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm89g\" (UID: \"d7bb31f1-07ea-4418-b744-fb09ba8f3f53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm89g" Jul 23 17:55:38.830374 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:38.830270 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7bb31f1-07ea-4418-b744-fb09ba8f3f53-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm89g\" (UID: \"d7bb31f1-07ea-4418-b744-fb09ba8f3f53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm89g" Jul 23 17:55:38.830374 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:38.830321 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7bb31f1-07ea-4418-b744-fb09ba8f3f53-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm89g\" (UID: \"d7bb31f1-07ea-4418-b744-fb09ba8f3f53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm89g" Jul 23 17:55:38.830374 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:38.830338 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7bb31f1-07ea-4418-b744-fb09ba8f3f53-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm89g\" (UID: \"d7bb31f1-07ea-4418-b744-fb09ba8f3f53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm89g" Jul 23 17:55:38.830534 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:38.830414 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7bb31f1-07ea-4418-b744-fb09ba8f3f53-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm89g\" (UID: \"d7bb31f1-07ea-4418-b744-fb09ba8f3f53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm89g" Jul 23 17:55:38.830721 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:38.830697 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7bb31f1-07ea-4418-b744-fb09ba8f3f53-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm89g\" (UID: \"d7bb31f1-07ea-4418-b744-fb09ba8f3f53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm89g" Jul 23 17:55:38.830812 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:38.830754 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7bb31f1-07ea-4418-b744-fb09ba8f3f53-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm89g\" (UID: \"d7bb31f1-07ea-4418-b744-fb09ba8f3f53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm89g" Jul 23 17:55:38.832583 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:38.832561 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7bb31f1-07ea-4418-b744-fb09ba8f3f53-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm89g\" (UID: \"d7bb31f1-07ea-4418-b744-fb09ba8f3f53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm89g" Jul 23 17:55:38.832817 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:38.832794 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7bb31f1-07ea-4418-b744-fb09ba8f3f53-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm89g\" (UID: \"d7bb31f1-07ea-4418-b744-fb09ba8f3f53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm89g" Jul 23 17:55:38.839407 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:38.839379 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6p82\" (UniqueName: \"kubernetes.io/projected/d7bb31f1-07ea-4418-b744-fb09ba8f3f53-kube-api-access-d6p82\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm89g\" (UID: \"d7bb31f1-07ea-4418-b744-fb09ba8f3f53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm89g" Jul 23 17:55:39.016752 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:39.016720 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm89g" Jul 23 17:55:39.156039 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:39.156012 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zm89g"] Jul 23 17:55:39.158176 ip-10-0-137-82 kubenswrapper[2565]: W0723 17:55:39.158144 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7bb31f1_07ea_4418_b744_fb09ba8f3f53.slice/crio-9d2a09a65b2d88349464b322aa96d09df73880c9d03eb0f0f4f5a4d1ccdab53b WatchSource:0}: Error finding container 9d2a09a65b2d88349464b322aa96d09df73880c9d03eb0f0f4f5a4d1ccdab53b: Status 404 returned error can't find the container with id 9d2a09a65b2d88349464b322aa96d09df73880c9d03eb0f0f4f5a4d1ccdab53b Jul 23 17:55:39.512883 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:39.512848 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm89g" event={"ID":"d7bb31f1-07ea-4418-b744-fb09ba8f3f53","Type":"ContainerStarted","Data":"8a196f3f454df2c4a166917adb1000dbe4569e691bdf86ec13dd27433b4c68b1"} Jul 23 17:55:39.513041 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:39.512903 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm89g" event={"ID":"d7bb31f1-07ea-4418-b744-fb09ba8f3f53","Type":"ContainerStarted","Data":"9d2a09a65b2d88349464b322aa96d09df73880c9d03eb0f0f4f5a4d1ccdab53b"} Jul 23 17:55:39.531690 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:39.531635 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm89g" podStartSLOduration=1.3270686 podStartE2EDuration="1.531619334s" totalImagesPullingTime="204.550734ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:55:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:55:39.529976886 +0000 UTC m=+683.608280240" watchObservedRunningTime="2026-07-23 17:55:39.531619334 +0000 UTC m=+683.609922686" Jul 23 17:55:49.016914 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:49.016879 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm89g" Jul 23 17:55:49.017288 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:49.017029 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm89g" Jul 23 17:55:49.030118 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:49.030093 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm89g" Jul 23 17:55:49.571356 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:55:49.571319 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm89g" Jul 23 17:59:16.367769 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:59:16.367688 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-m7k4j_1cca3236-5c90-4abb-9b9b-39e40b1903c3/console-operator/1.log" Jul 23 17:59:16.370528 ip-10-0-137-82 kubenswrapper[2565]: I0723 17:59:16.370505 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-m7k4j_1cca3236-5c90-4abb-9b9b-39e40b1903c3/console-operator/1.log" Jul 23 18:04:16.399257 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:04:16.399226 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-m7k4j_1cca3236-5c90-4abb-9b9b-39e40b1903c3/console-operator/1.log" Jul 23 18:04:16.404851 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:04:16.404824 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-m7k4j_1cca3236-5c90-4abb-9b9b-39e40b1903c3/console-operator/1.log" Jul 23 18:04:26.667796 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:04:26.667762 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2z86"] Jul 23 18:04:26.668186 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:04:26.667989 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2z86" podUID="1c6fdb49-b699-4fe8-9834-6a335376069d" containerName="manager" containerID="cri-o://21c89547b5e24206ac56b311abd199031c64247568373dbbccf8d17a8f4a2062" gracePeriod=10 Jul 23 18:04:29.823879 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:04:29.823855 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2z86" Jul 23 18:04:29.910427 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:04:29.910399 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c6fdb49-b699-4fe8-9834-6a335376069d-extensions-socket-volume\") pod \"1c6fdb49-b699-4fe8-9834-6a335376069d\" (UID: \"1c6fdb49-b699-4fe8-9834-6a335376069d\") " Jul 23 18:04:29.910588 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:04:29.910493 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttznl\" (UniqueName: \"kubernetes.io/projected/1c6fdb49-b699-4fe8-9834-6a335376069d-kube-api-access-ttznl\") pod \"1c6fdb49-b699-4fe8-9834-6a335376069d\" (UID: \"1c6fdb49-b699-4fe8-9834-6a335376069d\") " Jul 23 18:04:29.910769 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:04:29.910747 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c6fdb49-b699-4fe8-9834-6a335376069d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1c6fdb49-b699-4fe8-9834-6a335376069d" (UID: "1c6fdb49-b699-4fe8-9834-6a335376069d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:04:29.912575 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:04:29.912553 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c6fdb49-b699-4fe8-9834-6a335376069d-kube-api-access-ttznl" (OuterVolumeSpecName: "kube-api-access-ttznl") pod "1c6fdb49-b699-4fe8-9834-6a335376069d" (UID: "1c6fdb49-b699-4fe8-9834-6a335376069d"). InnerVolumeSpecName "kube-api-access-ttznl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:04:30.011700 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:04:30.011618 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttznl\" (UniqueName: \"kubernetes.io/projected/1c6fdb49-b699-4fe8-9834-6a335376069d-kube-api-access-ttznl\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:04:30.011700 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:04:30.011644 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c6fdb49-b699-4fe8-9834-6a335376069d-extensions-socket-volume\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:04:30.041445 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:04:30.041407 2565 generic.go:356] "Generic (PLEG): container finished" podID="1c6fdb49-b699-4fe8-9834-6a335376069d" containerID="21c89547b5e24206ac56b311abd199031c64247568373dbbccf8d17a8f4a2062" exitCode=0 Jul 23 18:04:30.041445 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:04:30.041447 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2z86" Jul 23 18:04:30.041688 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:04:30.041490 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2z86" event={"ID":"1c6fdb49-b699-4fe8-9834-6a335376069d","Type":"ContainerDied","Data":"21c89547b5e24206ac56b311abd199031c64247568373dbbccf8d17a8f4a2062"} Jul 23 18:04:30.041688 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:04:30.041536 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2z86" event={"ID":"1c6fdb49-b699-4fe8-9834-6a335376069d","Type":"ContainerDied","Data":"5cd90887032575f1097f0b5461d51e3ff89fbf105ae345a01bb61ada436cca81"} Jul 23 18:04:30.041688 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:04:30.041560 2565 scope.go:117] "RemoveContainer" containerID="21c89547b5e24206ac56b311abd199031c64247568373dbbccf8d17a8f4a2062" Jul 23 18:04:30.050883 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:04:30.050868 2565 scope.go:117] "RemoveContainer" containerID="21c89547b5e24206ac56b311abd199031c64247568373dbbccf8d17a8f4a2062" Jul 23 18:04:30.051123 ip-10-0-137-82 kubenswrapper[2565]: E0723 18:04:30.051105 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21c89547b5e24206ac56b311abd199031c64247568373dbbccf8d17a8f4a2062\": container with ID starting with 21c89547b5e24206ac56b311abd199031c64247568373dbbccf8d17a8f4a2062 not found: ID does not exist" containerID="21c89547b5e24206ac56b311abd199031c64247568373dbbccf8d17a8f4a2062" Jul 23 18:04:30.051176 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:04:30.051130 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21c89547b5e24206ac56b311abd199031c64247568373dbbccf8d17a8f4a2062"} err="failed to get container status \"21c89547b5e24206ac56b311abd199031c64247568373dbbccf8d17a8f4a2062\": rpc error: code = NotFound desc = could not find container \"21c89547b5e24206ac56b311abd199031c64247568373dbbccf8d17a8f4a2062\": container with ID starting with 21c89547b5e24206ac56b311abd199031c64247568373dbbccf8d17a8f4a2062 not found: ID does not exist" Jul 23 18:04:30.070756 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:04:30.070059 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2z86"] Jul 23 18:04:30.072591 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:04:30.072564 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b2z86"] Jul 23 18:04:30.406428 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:04:30.406391 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c6fdb49-b699-4fe8-9834-6a335376069d" path="/var/lib/kubelet/pods/1c6fdb49-b699-4fe8-9834-6a335376069d/volumes" Jul 23 18:05:27.846035 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:05:27.845996 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9874m"] Jul 23 18:05:27.846537 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:05:27.846520 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c6fdb49-b699-4fe8-9834-6a335376069d" containerName="manager" Jul 23 18:05:27.846584 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:05:27.846540 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c6fdb49-b699-4fe8-9834-6a335376069d" containerName="manager" Jul 23 18:05:27.846672 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:05:27.846660 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c6fdb49-b699-4fe8-9834-6a335376069d" containerName="manager" Jul 23 18:05:27.851015 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:05:27.850999 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9874m" Jul 23 18:05:27.853826 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:05:27.853799 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jpgwl\"" Jul 23 18:05:27.854114 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:05:27.854098 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 18:05:27.855914 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:05:27.855896 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 18:05:27.867133 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:05:27.867112 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9874m"] Jul 23 18:05:28.000425 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:05:28.000377 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf6df2af-6e72-4bd7-aa51-04dcd0fc1bbe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9874m\" (UID: \"cf6df2af-6e72-4bd7-aa51-04dcd0fc1bbe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9874m" Jul 23 18:05:28.000625 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:05:28.000453 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvq5r\" (UniqueName: \"kubernetes.io/projected/cf6df2af-6e72-4bd7-aa51-04dcd0fc1bbe-kube-api-access-cvq5r\") pod \"kuadrant-operator-controller-manager-86878c769f-9874m\" (UID: \"cf6df2af-6e72-4bd7-aa51-04dcd0fc1bbe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9874m" Jul 23 18:05:28.101783 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:05:28.101687 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf6df2af-6e72-4bd7-aa51-04dcd0fc1bbe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9874m\" (UID: \"cf6df2af-6e72-4bd7-aa51-04dcd0fc1bbe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9874m" Jul 23 18:05:28.101783 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:05:28.101745 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvq5r\" (UniqueName: \"kubernetes.io/projected/cf6df2af-6e72-4bd7-aa51-04dcd0fc1bbe-kube-api-access-cvq5r\") pod \"kuadrant-operator-controller-manager-86878c769f-9874m\" (UID: \"cf6df2af-6e72-4bd7-aa51-04dcd0fc1bbe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9874m" Jul 23 18:05:28.102090 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:05:28.102068 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf6df2af-6e72-4bd7-aa51-04dcd0fc1bbe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9874m\" (UID: \"cf6df2af-6e72-4bd7-aa51-04dcd0fc1bbe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9874m" Jul 23 18:05:28.112069 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:05:28.112042 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvq5r\" (UniqueName: \"kubernetes.io/projected/cf6df2af-6e72-4bd7-aa51-04dcd0fc1bbe-kube-api-access-cvq5r\") pod \"kuadrant-operator-controller-manager-86878c769f-9874m\" (UID: \"cf6df2af-6e72-4bd7-aa51-04dcd0fc1bbe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9874m" Jul 23 18:05:28.161642 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:05:28.161584 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9874m" Jul 23 18:05:28.323021 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:05:28.322984 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9874m"] Jul 23 18:05:28.325978 ip-10-0-137-82 kubenswrapper[2565]: W0723 18:05:28.325947 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf6df2af_6e72_4bd7_aa51_04dcd0fc1bbe.slice/crio-cdf3482281b9cd656ed918a47e4da66ceb2b45149cb6e2b0c053674adc821844 WatchSource:0}: Error finding container cdf3482281b9cd656ed918a47e4da66ceb2b45149cb6e2b0c053674adc821844: Status 404 returned error can't find the container with id cdf3482281b9cd656ed918a47e4da66ceb2b45149cb6e2b0c053674adc821844 Jul 23 18:05:29.323405 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:05:29.323357 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9874m" event={"ID":"cf6df2af-6e72-4bd7-aa51-04dcd0fc1bbe","Type":"ContainerStarted","Data":"ad061ec1b2e3928c7c89fd7edabe6ec63a370937a73c207df40f3e94e3b8e28b"} Jul 23 18:05:29.323869 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:05:29.323428 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9874m" Jul 23 18:05:29.323869 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:05:29.323444 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9874m" event={"ID":"cf6df2af-6e72-4bd7-aa51-04dcd0fc1bbe","Type":"ContainerStarted","Data":"cdf3482281b9cd656ed918a47e4da66ceb2b45149cb6e2b0c053674adc821844"} Jul 23 18:05:29.355088 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:05:29.355039 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9874m" podStartSLOduration=2.355023316 podStartE2EDuration="2.355023316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:05:29.353940707 +0000 UTC m=+1273.432244063" watchObservedRunningTime="2026-07-23 18:05:29.355023316 +0000 UTC m=+1273.433326669" Jul 23 18:05:40.330960 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:05:40.330925 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9874m" Jul 23 18:09:16.429702 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:09:16.429675 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-m7k4j_1cca3236-5c90-4abb-9b9b-39e40b1903c3/console-operator/1.log" Jul 23 18:09:16.434893 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:09:16.434871 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-m7k4j_1cca3236-5c90-4abb-9b9b-39e40b1903c3/console-operator/1.log" Jul 23 18:12:32.614340 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:12:32.614306 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-60a07bef-688c88fd56-246mf"] Jul 23 18:12:32.617788 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:12:32.617771 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a07bef-688c88fd56-246mf" Jul 23 18:12:32.620943 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:12:32.620920 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-60a07bef-serving-cert\"" Jul 23 18:12:32.639319 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:12:32.639290 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-60a07bef-688c88fd56-246mf"] Jul 23 18:12:32.724932 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:12:32.724887 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mnrw\" (UniqueName: \"kubernetes.io/projected/786e4830-7c0b-46ce-a70b-5a699f385c8d-kube-api-access-2mnrw\") pod \"maas-api-e2e-ait-60a07bef-688c88fd56-246mf\" (UID: \"786e4830-7c0b-46ce-a70b-5a699f385c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a07bef-688c88fd56-246mf" Jul 23 18:12:32.725126 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:12:32.724982 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/786e4830-7c0b-46ce-a70b-5a699f385c8d-maas-api-tls\") pod \"maas-api-e2e-ait-60a07bef-688c88fd56-246mf\" (UID: \"786e4830-7c0b-46ce-a70b-5a699f385c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a07bef-688c88fd56-246mf" Jul 23 18:12:32.725126 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:12:32.725027 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/786e4830-7c0b-46ce-a70b-5a699f385c8d-openshift-service-ca\") pod \"maas-api-e2e-ait-60a07bef-688c88fd56-246mf\" (UID: \"786e4830-7c0b-46ce-a70b-5a699f385c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a07bef-688c88fd56-246mf" Jul 23 18:12:32.825909 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:12:32.825865 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/786e4830-7c0b-46ce-a70b-5a699f385c8d-maas-api-tls\") pod \"maas-api-e2e-ait-60a07bef-688c88fd56-246mf\" (UID: \"786e4830-7c0b-46ce-a70b-5a699f385c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a07bef-688c88fd56-246mf" Jul 23 18:12:32.826110 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:12:32.825921 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/786e4830-7c0b-46ce-a70b-5a699f385c8d-openshift-service-ca\") pod \"maas-api-e2e-ait-60a07bef-688c88fd56-246mf\" (UID: \"786e4830-7c0b-46ce-a70b-5a699f385c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a07bef-688c88fd56-246mf" Jul 23 18:12:32.826110 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:12:32.825968 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mnrw\" (UniqueName: \"kubernetes.io/projected/786e4830-7c0b-46ce-a70b-5a699f385c8d-kube-api-access-2mnrw\") pod \"maas-api-e2e-ait-60a07bef-688c88fd56-246mf\" (UID: \"786e4830-7c0b-46ce-a70b-5a699f385c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a07bef-688c88fd56-246mf" Jul 23 18:12:32.826110 ip-10-0-137-82 kubenswrapper[2565]: E0723 18:12:32.826029 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-60a07bef-serving-cert: secret "maas-api-e2e-ait-60a07bef-serving-cert" not found Jul 23 18:12:32.826275 ip-10-0-137-82 kubenswrapper[2565]: E0723 18:12:32.826114 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/786e4830-7c0b-46ce-a70b-5a699f385c8d-maas-api-tls podName:786e4830-7c0b-46ce-a70b-5a699f385c8d nodeName:}" failed. No retries permitted until 2026-07-23 18:12:33.326096221 +0000 UTC m=+1697.404399552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/786e4830-7c0b-46ce-a70b-5a699f385c8d-maas-api-tls") pod "maas-api-e2e-ait-60a07bef-688c88fd56-246mf" (UID: "786e4830-7c0b-46ce-a70b-5a699f385c8d") : secret "maas-api-e2e-ait-60a07bef-serving-cert" not found Jul 23 18:12:32.826818 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:12:32.826796 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/786e4830-7c0b-46ce-a70b-5a699f385c8d-openshift-service-ca\") pod \"maas-api-e2e-ait-60a07bef-688c88fd56-246mf\" (UID: \"786e4830-7c0b-46ce-a70b-5a699f385c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a07bef-688c88fd56-246mf" Jul 23 18:12:32.839313 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:12:32.839286 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mnrw\" (UniqueName: \"kubernetes.io/projected/786e4830-7c0b-46ce-a70b-5a699f385c8d-kube-api-access-2mnrw\") pod \"maas-api-e2e-ait-60a07bef-688c88fd56-246mf\" (UID: \"786e4830-7c0b-46ce-a70b-5a699f385c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a07bef-688c88fd56-246mf" Jul 23 18:12:33.330700 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:12:33.330661 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/786e4830-7c0b-46ce-a70b-5a699f385c8d-maas-api-tls\") pod \"maas-api-e2e-ait-60a07bef-688c88fd56-246mf\" (UID: \"786e4830-7c0b-46ce-a70b-5a699f385c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a07bef-688c88fd56-246mf" Jul 23 18:12:33.333091 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:12:33.333063 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/786e4830-7c0b-46ce-a70b-5a699f385c8d-maas-api-tls\") pod \"maas-api-e2e-ait-60a07bef-688c88fd56-246mf\" (UID: \"786e4830-7c0b-46ce-a70b-5a699f385c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a07bef-688c88fd56-246mf" Jul 23 18:12:33.531248 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:12:33.531212 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a07bef-688c88fd56-246mf" Jul 23 18:12:33.679626 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:12:33.679585 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-60a07bef-688c88fd56-246mf"] Jul 23 18:12:33.682447 ip-10-0-137-82 kubenswrapper[2565]: W0723 18:12:33.682414 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod786e4830_7c0b_46ce_a70b_5a699f385c8d.slice/crio-ee26ed92f06ea90e30f8f08d06a1ac05fab5920734f550f1b09f10f698c769db WatchSource:0}: Error finding container ee26ed92f06ea90e30f8f08d06a1ac05fab5920734f550f1b09f10f698c769db: Status 404 returned error can't find the container with id ee26ed92f06ea90e30f8f08d06a1ac05fab5920734f550f1b09f10f698c769db Jul 23 18:12:33.683796 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:12:33.683777 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 18:12:34.344294 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:12:34.344260 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a07bef-688c88fd56-246mf" event={"ID":"786e4830-7c0b-46ce-a70b-5a699f385c8d","Type":"ContainerStarted","Data":"d65cba70b262621203b5ef031abb7aaa7b685eef8243d02aa42cc7fc04371117"} Jul 23 18:12:34.344461 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:12:34.344318 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a07bef-688c88fd56-246mf" Jul 23 18:12:34.344461 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:12:34.344335 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a07bef-688c88fd56-246mf" event={"ID":"786e4830-7c0b-46ce-a70b-5a699f385c8d","Type":"ContainerStarted","Data":"ee26ed92f06ea90e30f8f08d06a1ac05fab5920734f550f1b09f10f698c769db"} Jul 23 18:12:34.371108 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:12:34.371055 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a07bef-688c88fd56-246mf" podStartSLOduration=1.852970457 podStartE2EDuration="2.371040652s" totalImagesPullingTime="518.070195ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:12:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:12:34.370927022 +0000 UTC m=+1698.449230374" watchObservedRunningTime="2026-07-23 18:12:34.371040652 +0000 UTC m=+1698.449344004" Jul 23 18:12:40.354874 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:12:40.354796 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a07bef-688c88fd56-246mf" Jul 23 18:13:08.537464 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:08.537425 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-60a07bef-688c88fd56-246mf"] Jul 23 18:13:08.538092 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:08.537694 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a07bef-688c88fd56-246mf" podUID="786e4830-7c0b-46ce-a70b-5a699f385c8d" containerName="maas-api" containerID="cri-o://d65cba70b262621203b5ef031abb7aaa7b685eef8243d02aa42cc7fc04371117" gracePeriod=30 Jul 23 18:13:08.792807 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:08.792751 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a07bef-688c88fd56-246mf" Jul 23 18:13:08.821534 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:08.821495 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mnrw\" (UniqueName: \"kubernetes.io/projected/786e4830-7c0b-46ce-a70b-5a699f385c8d-kube-api-access-2mnrw\") pod \"786e4830-7c0b-46ce-a70b-5a699f385c8d\" (UID: \"786e4830-7c0b-46ce-a70b-5a699f385c8d\") " Jul 23 18:13:08.821713 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:08.821577 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/786e4830-7c0b-46ce-a70b-5a699f385c8d-openshift-service-ca\") pod \"786e4830-7c0b-46ce-a70b-5a699f385c8d\" (UID: \"786e4830-7c0b-46ce-a70b-5a699f385c8d\") " Jul 23 18:13:08.821713 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:08.821688 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/786e4830-7c0b-46ce-a70b-5a699f385c8d-maas-api-tls\") pod \"786e4830-7c0b-46ce-a70b-5a699f385c8d\" (UID: \"786e4830-7c0b-46ce-a70b-5a699f385c8d\") " Jul 23 18:13:08.822156 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:08.822123 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/786e4830-7c0b-46ce-a70b-5a699f385c8d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "786e4830-7c0b-46ce-a70b-5a699f385c8d" (UID: "786e4830-7c0b-46ce-a70b-5a699f385c8d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:13:08.824071 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:08.824035 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/786e4830-7c0b-46ce-a70b-5a699f385c8d-kube-api-access-2mnrw" (OuterVolumeSpecName: "kube-api-access-2mnrw") pod "786e4830-7c0b-46ce-a70b-5a699f385c8d" (UID: "786e4830-7c0b-46ce-a70b-5a699f385c8d"). InnerVolumeSpecName "kube-api-access-2mnrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:13:08.824510 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:08.824482 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/786e4830-7c0b-46ce-a70b-5a699f385c8d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "786e4830-7c0b-46ce-a70b-5a699f385c8d" (UID: "786e4830-7c0b-46ce-a70b-5a699f385c8d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:13:08.922510 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:08.922471 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/786e4830-7c0b-46ce-a70b-5a699f385c8d-openshift-service-ca\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:13:08.922510 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:08.922508 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/786e4830-7c0b-46ce-a70b-5a699f385c8d-maas-api-tls\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:13:08.922726 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:08.922520 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mnrw\" (UniqueName: \"kubernetes.io/projected/786e4830-7c0b-46ce-a70b-5a699f385c8d-kube-api-access-2mnrw\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:13:09.514158 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:09.514121 2565 generic.go:356] "Generic (PLEG): container finished" podID="786e4830-7c0b-46ce-a70b-5a699f385c8d" containerID="d65cba70b262621203b5ef031abb7aaa7b685eef8243d02aa42cc7fc04371117" exitCode=0 Jul 23 18:13:09.514338 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:09.514165 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a07bef-688c88fd56-246mf" Jul 23 18:13:09.514338 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:09.514201 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a07bef-688c88fd56-246mf" event={"ID":"786e4830-7c0b-46ce-a70b-5a699f385c8d","Type":"ContainerDied","Data":"d65cba70b262621203b5ef031abb7aaa7b685eef8243d02aa42cc7fc04371117"} Jul 23 18:13:09.514338 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:09.514248 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a07bef-688c88fd56-246mf" event={"ID":"786e4830-7c0b-46ce-a70b-5a699f385c8d","Type":"ContainerDied","Data":"ee26ed92f06ea90e30f8f08d06a1ac05fab5920734f550f1b09f10f698c769db"} Jul 23 18:13:09.514338 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:09.514269 2565 scope.go:117] "RemoveContainer" containerID="d65cba70b262621203b5ef031abb7aaa7b685eef8243d02aa42cc7fc04371117" Jul 23 18:13:09.523513 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:09.523496 2565 scope.go:117] "RemoveContainer" containerID="d65cba70b262621203b5ef031abb7aaa7b685eef8243d02aa42cc7fc04371117" Jul 23 18:13:09.523807 ip-10-0-137-82 kubenswrapper[2565]: E0723 18:13:09.523779 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d65cba70b262621203b5ef031abb7aaa7b685eef8243d02aa42cc7fc04371117\": container with ID starting with d65cba70b262621203b5ef031abb7aaa7b685eef8243d02aa42cc7fc04371117 not found: ID does not exist" containerID="d65cba70b262621203b5ef031abb7aaa7b685eef8243d02aa42cc7fc04371117" Jul 23 18:13:09.523909 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:09.523809 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d65cba70b262621203b5ef031abb7aaa7b685eef8243d02aa42cc7fc04371117"} err="failed to get container status \"d65cba70b262621203b5ef031abb7aaa7b685eef8243d02aa42cc7fc04371117\": rpc error: code = NotFound desc = could not find container \"d65cba70b262621203b5ef031abb7aaa7b685eef8243d02aa42cc7fc04371117\": container with ID starting with d65cba70b262621203b5ef031abb7aaa7b685eef8243d02aa42cc7fc04371117 not found: ID does not exist" Jul 23 18:13:09.537108 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:09.537080 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-60a07bef-688c88fd56-246mf"] Jul 23 18:13:09.543816 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:09.543793 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-60a07bef-688c88fd56-246mf"] Jul 23 18:13:10.405987 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:10.405953 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="786e4830-7c0b-46ce-a70b-5a699f385c8d" path="/var/lib/kubelet/pods/786e4830-7c0b-46ce-a70b-5a699f385c8d/volumes" Jul 23 18:13:19.961110 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:19.961078 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp"] Jul 23 18:13:19.961490 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:19.961466 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="786e4830-7c0b-46ce-a70b-5a699f385c8d" containerName="maas-api" Jul 23 18:13:19.961490 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:19.961477 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="786e4830-7c0b-46ce-a70b-5a699f385c8d" containerName="maas-api" Jul 23 18:13:19.961566 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:19.961555 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="786e4830-7c0b-46ce-a70b-5a699f385c8d" containerName="maas-api" Jul 23 18:13:19.964276 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:19.964262 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp" Jul 23 18:13:19.967283 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:19.967262 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-69ff141b-serving-cert\"" Jul 23 18:13:19.977052 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:19.977027 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp"] Jul 23 18:13:20.023496 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:20.023463 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4tfj\" (UniqueName: \"kubernetes.io/projected/15aa5d05-a197-49e5-9da8-b34b53da38ee-kube-api-access-n4tfj\") pod \"maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp\" (UID: \"15aa5d05-a197-49e5-9da8-b34b53da38ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp" Jul 23 18:13:20.023705 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:20.023593 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15aa5d05-a197-49e5-9da8-b34b53da38ee-maas-api-tls\") pod \"maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp\" (UID: \"15aa5d05-a197-49e5-9da8-b34b53da38ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp" Jul 23 18:13:20.023705 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:20.023667 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15aa5d05-a197-49e5-9da8-b34b53da38ee-openshift-service-ca\") pod \"maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp\" (UID: \"15aa5d05-a197-49e5-9da8-b34b53da38ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp" Jul 23 18:13:20.124142 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:20.124103 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4tfj\" (UniqueName: \"kubernetes.io/projected/15aa5d05-a197-49e5-9da8-b34b53da38ee-kube-api-access-n4tfj\") pod \"maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp\" (UID: \"15aa5d05-a197-49e5-9da8-b34b53da38ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp" Jul 23 18:13:20.124313 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:20.124185 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15aa5d05-a197-49e5-9da8-b34b53da38ee-maas-api-tls\") pod \"maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp\" (UID: \"15aa5d05-a197-49e5-9da8-b34b53da38ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp" Jul 23 18:13:20.124313 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:20.124226 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15aa5d05-a197-49e5-9da8-b34b53da38ee-openshift-service-ca\") pod \"maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp\" (UID: \"15aa5d05-a197-49e5-9da8-b34b53da38ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp" Jul 23 18:13:20.124405 ip-10-0-137-82 kubenswrapper[2565]: E0723 18:13:20.124329 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-69ff141b-serving-cert: secret "maas-api-e2e-migrate-69ff141b-serving-cert" not found Jul 23 18:13:20.124449 ip-10-0-137-82 kubenswrapper[2565]: E0723 18:13:20.124407 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15aa5d05-a197-49e5-9da8-b34b53da38ee-maas-api-tls podName:15aa5d05-a197-49e5-9da8-b34b53da38ee nodeName:}" failed. No retries permitted until 2026-07-23 18:13:20.624387732 +0000 UTC m=+1744.702691068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/15aa5d05-a197-49e5-9da8-b34b53da38ee-maas-api-tls") pod "maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp" (UID: "15aa5d05-a197-49e5-9da8-b34b53da38ee") : secret "maas-api-e2e-migrate-69ff141b-serving-cert" not found Jul 23 18:13:20.124950 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:20.124925 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15aa5d05-a197-49e5-9da8-b34b53da38ee-openshift-service-ca\") pod \"maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp\" (UID: \"15aa5d05-a197-49e5-9da8-b34b53da38ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp" Jul 23 18:13:20.135082 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:20.135058 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4tfj\" (UniqueName: \"kubernetes.io/projected/15aa5d05-a197-49e5-9da8-b34b53da38ee-kube-api-access-n4tfj\") pod \"maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp\" (UID: \"15aa5d05-a197-49e5-9da8-b34b53da38ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp" Jul 23 18:13:20.628698 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:20.628663 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15aa5d05-a197-49e5-9da8-b34b53da38ee-maas-api-tls\") pod \"maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp\" (UID: \"15aa5d05-a197-49e5-9da8-b34b53da38ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp" Jul 23 18:13:20.631259 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:20.631232 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15aa5d05-a197-49e5-9da8-b34b53da38ee-maas-api-tls\") pod \"maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp\" (UID: \"15aa5d05-a197-49e5-9da8-b34b53da38ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp" Jul 23 18:13:20.874957 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:20.874919 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp" Jul 23 18:13:21.012800 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:21.012769 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp"] Jul 23 18:13:21.014800 ip-10-0-137-82 kubenswrapper[2565]: W0723 18:13:21.014773 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15aa5d05_a197_49e5_9da8_b34b53da38ee.slice/crio-a00b0526251060a45e2dd0ce47c5fc8a17fcdca680ff612c5df08aaf161f878f WatchSource:0}: Error finding container a00b0526251060a45e2dd0ce47c5fc8a17fcdca680ff612c5df08aaf161f878f: Status 404 returned error can't find the container with id a00b0526251060a45e2dd0ce47c5fc8a17fcdca680ff612c5df08aaf161f878f Jul 23 18:13:21.578795 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:21.578770 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp" event={"ID":"15aa5d05-a197-49e5-9da8-b34b53da38ee","Type":"ContainerStarted","Data":"cc44c0d833ef3e46452d96cdeb7d57efb7e8f3bcf9d5406764783f8aaa8aa9b2"} Jul 23 18:13:21.578906 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:21.578817 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp" event={"ID":"15aa5d05-a197-49e5-9da8-b34b53da38ee","Type":"ContainerStarted","Data":"a00b0526251060a45e2dd0ce47c5fc8a17fcdca680ff612c5df08aaf161f878f"} Jul 23 18:13:21.578906 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:21.578861 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp" Jul 23 18:13:21.600661 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:21.600620 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp" podStartSLOduration=2.10642589 podStartE2EDuration="2.600580914s" totalImagesPullingTime="494.155024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:13:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:13:21.599379219 +0000 UTC m=+1745.677682595" watchObservedRunningTime="2026-07-23 18:13:21.600580914 +0000 UTC m=+1745.678884266" Jul 23 18:13:27.589088 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:27.589060 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp" Jul 23 18:13:55.425118 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:55.425079 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp"] Jul 23 18:13:55.425658 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:55.425325 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp" podUID="15aa5d05-a197-49e5-9da8-b34b53da38ee" containerName="maas-api" containerID="cri-o://cc44c0d833ef3e46452d96cdeb7d57efb7e8f3bcf9d5406764783f8aaa8aa9b2" gracePeriod=30 Jul 23 18:13:55.681017 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:55.680961 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp" Jul 23 18:13:55.737501 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:55.737467 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4tfj\" (UniqueName: \"kubernetes.io/projected/15aa5d05-a197-49e5-9da8-b34b53da38ee-kube-api-access-n4tfj\") pod \"15aa5d05-a197-49e5-9da8-b34b53da38ee\" (UID: \"15aa5d05-a197-49e5-9da8-b34b53da38ee\") " Jul 23 18:13:55.737501 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:55.737509 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15aa5d05-a197-49e5-9da8-b34b53da38ee-maas-api-tls\") pod \"15aa5d05-a197-49e5-9da8-b34b53da38ee\" (UID: \"15aa5d05-a197-49e5-9da8-b34b53da38ee\") " Jul 23 18:13:55.737726 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:55.737560 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15aa5d05-a197-49e5-9da8-b34b53da38ee-openshift-service-ca\") pod \"15aa5d05-a197-49e5-9da8-b34b53da38ee\" (UID: \"15aa5d05-a197-49e5-9da8-b34b53da38ee\") " Jul 23 18:13:55.738094 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:55.738054 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15aa5d05-a197-49e5-9da8-b34b53da38ee-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "15aa5d05-a197-49e5-9da8-b34b53da38ee" (UID: "15aa5d05-a197-49e5-9da8-b34b53da38ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:13:55.739663 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:55.739638 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15aa5d05-a197-49e5-9da8-b34b53da38ee-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "15aa5d05-a197-49e5-9da8-b34b53da38ee" (UID: "15aa5d05-a197-49e5-9da8-b34b53da38ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:13:55.739763 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:55.739707 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15aa5d05-a197-49e5-9da8-b34b53da38ee-kube-api-access-n4tfj" (OuterVolumeSpecName: "kube-api-access-n4tfj") pod "15aa5d05-a197-49e5-9da8-b34b53da38ee" (UID: "15aa5d05-a197-49e5-9da8-b34b53da38ee"). InnerVolumeSpecName "kube-api-access-n4tfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:13:55.749576 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:55.749550 2565 generic.go:356] "Generic (PLEG): container finished" podID="15aa5d05-a197-49e5-9da8-b34b53da38ee" containerID="cc44c0d833ef3e46452d96cdeb7d57efb7e8f3bcf9d5406764783f8aaa8aa9b2" exitCode=0 Jul 23 18:13:55.749714 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:55.749623 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp" Jul 23 18:13:55.749714 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:55.749640 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp" event={"ID":"15aa5d05-a197-49e5-9da8-b34b53da38ee","Type":"ContainerDied","Data":"cc44c0d833ef3e46452d96cdeb7d57efb7e8f3bcf9d5406764783f8aaa8aa9b2"} Jul 23 18:13:55.749714 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:55.749695 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp" event={"ID":"15aa5d05-a197-49e5-9da8-b34b53da38ee","Type":"ContainerDied","Data":"a00b0526251060a45e2dd0ce47c5fc8a17fcdca680ff612c5df08aaf161f878f"} Jul 23 18:13:55.749856 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:55.749717 2565 scope.go:117] "RemoveContainer" containerID="cc44c0d833ef3e46452d96cdeb7d57efb7e8f3bcf9d5406764783f8aaa8aa9b2" Jul 23 18:13:55.759269 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:55.759249 2565 scope.go:117] "RemoveContainer" containerID="cc44c0d833ef3e46452d96cdeb7d57efb7e8f3bcf9d5406764783f8aaa8aa9b2" Jul 23 18:13:55.759547 ip-10-0-137-82 kubenswrapper[2565]: E0723 18:13:55.759519 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc44c0d833ef3e46452d96cdeb7d57efb7e8f3bcf9d5406764783f8aaa8aa9b2\": container with ID starting with cc44c0d833ef3e46452d96cdeb7d57efb7e8f3bcf9d5406764783f8aaa8aa9b2 not found: ID does not exist" containerID="cc44c0d833ef3e46452d96cdeb7d57efb7e8f3bcf9d5406764783f8aaa8aa9b2" Jul 23 18:13:55.759648 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:55.759560 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc44c0d833ef3e46452d96cdeb7d57efb7e8f3bcf9d5406764783f8aaa8aa9b2"} err="failed to get container status \"cc44c0d833ef3e46452d96cdeb7d57efb7e8f3bcf9d5406764783f8aaa8aa9b2\": rpc error: code = NotFound desc = could not find container \"cc44c0d833ef3e46452d96cdeb7d57efb7e8f3bcf9d5406764783f8aaa8aa9b2\": container with ID starting with cc44c0d833ef3e46452d96cdeb7d57efb7e8f3bcf9d5406764783f8aaa8aa9b2 not found: ID does not exist" Jul 23 18:13:55.773130 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:55.773104 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp"] Jul 23 18:13:55.776792 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:55.776769 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-69ff141b-5bdb95d7bc-x7bqp"] Jul 23 18:13:55.839217 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:55.839182 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4tfj\" (UniqueName: \"kubernetes.io/projected/15aa5d05-a197-49e5-9da8-b34b53da38ee-kube-api-access-n4tfj\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:13:55.839217 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:55.839219 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15aa5d05-a197-49e5-9da8-b34b53da38ee-maas-api-tls\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:13:55.839391 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:55.839234 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15aa5d05-a197-49e5-9da8-b34b53da38ee-openshift-service-ca\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:13:56.406706 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:13:56.406670 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15aa5d05-a197-49e5-9da8-b34b53da38ee" path="/var/lib/kubelet/pods/15aa5d05-a197-49e5-9da8-b34b53da38ee/volumes" Jul 23 18:14:03.352227 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:03.352188 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f954644c-7646d9f4b-j2q89"] Jul 23 18:14:03.352805 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:03.352788 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15aa5d05-a197-49e5-9da8-b34b53da38ee" containerName="maas-api" Jul 23 18:14:03.352856 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:03.352807 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="15aa5d05-a197-49e5-9da8-b34b53da38ee" containerName="maas-api" Jul 23 18:14:03.352911 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:03.352899 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="15aa5d05-a197-49e5-9da8-b34b53da38ee" containerName="maas-api" Jul 23 18:14:03.356300 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:03.356282 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f954644c-7646d9f4b-j2q89" Jul 23 18:14:03.361891 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:03.361871 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f954644c-serving-cert\"" Jul 23 18:14:03.375296 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:03.375271 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f954644c-7646d9f4b-j2q89"] Jul 23 18:14:03.408688 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:03.408648 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23f022d5-fa7e-46ce-aab7-d2f4008a2ef1-maas-api-tls\") pod \"maas-api-e2e-ait-f954644c-7646d9f4b-j2q89\" (UID: \"23f022d5-fa7e-46ce-aab7-d2f4008a2ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f954644c-7646d9f4b-j2q89" Jul 23 18:14:03.408688 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:03.408683 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23f022d5-fa7e-46ce-aab7-d2f4008a2ef1-openshift-service-ca\") pod \"maas-api-e2e-ait-f954644c-7646d9f4b-j2q89\" (UID: \"23f022d5-fa7e-46ce-aab7-d2f4008a2ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f954644c-7646d9f4b-j2q89" Jul 23 18:14:03.408887 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:03.408812 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfm4r\" (UniqueName: \"kubernetes.io/projected/23f022d5-fa7e-46ce-aab7-d2f4008a2ef1-kube-api-access-pfm4r\") pod \"maas-api-e2e-ait-f954644c-7646d9f4b-j2q89\" (UID: \"23f022d5-fa7e-46ce-aab7-d2f4008a2ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f954644c-7646d9f4b-j2q89" Jul 23 18:14:03.509685 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:03.509632 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23f022d5-fa7e-46ce-aab7-d2f4008a2ef1-maas-api-tls\") pod \"maas-api-e2e-ait-f954644c-7646d9f4b-j2q89\" (UID: \"23f022d5-fa7e-46ce-aab7-d2f4008a2ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f954644c-7646d9f4b-j2q89" Jul 23 18:14:03.509685 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:03.509682 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23f022d5-fa7e-46ce-aab7-d2f4008a2ef1-openshift-service-ca\") pod \"maas-api-e2e-ait-f954644c-7646d9f4b-j2q89\" (UID: \"23f022d5-fa7e-46ce-aab7-d2f4008a2ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f954644c-7646d9f4b-j2q89" Jul 23 18:14:03.509945 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:03.509744 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfm4r\" (UniqueName: \"kubernetes.io/projected/23f022d5-fa7e-46ce-aab7-d2f4008a2ef1-kube-api-access-pfm4r\") pod \"maas-api-e2e-ait-f954644c-7646d9f4b-j2q89\" (UID: \"23f022d5-fa7e-46ce-aab7-d2f4008a2ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f954644c-7646d9f4b-j2q89" Jul 23 18:14:03.509945 ip-10-0-137-82 kubenswrapper[2565]: E0723 18:14:03.509809 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f954644c-serving-cert: secret "maas-api-e2e-ait-f954644c-serving-cert" not found Jul 23 18:14:03.509945 ip-10-0-137-82 kubenswrapper[2565]: E0723 18:14:03.509895 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23f022d5-fa7e-46ce-aab7-d2f4008a2ef1-maas-api-tls podName:23f022d5-fa7e-46ce-aab7-d2f4008a2ef1 nodeName:}" failed. No retries permitted until 2026-07-23 18:14:04.009872566 +0000 UTC m=+1788.088175899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/23f022d5-fa7e-46ce-aab7-d2f4008a2ef1-maas-api-tls") pod "maas-api-e2e-ait-f954644c-7646d9f4b-j2q89" (UID: "23f022d5-fa7e-46ce-aab7-d2f4008a2ef1") : secret "maas-api-e2e-ait-f954644c-serving-cert" not found Jul 23 18:14:03.510363 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:03.510344 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23f022d5-fa7e-46ce-aab7-d2f4008a2ef1-openshift-service-ca\") pod \"maas-api-e2e-ait-f954644c-7646d9f4b-j2q89\" (UID: \"23f022d5-fa7e-46ce-aab7-d2f4008a2ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f954644c-7646d9f4b-j2q89" Jul 23 18:14:03.543202 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:03.543169 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfm4r\" (UniqueName: \"kubernetes.io/projected/23f022d5-fa7e-46ce-aab7-d2f4008a2ef1-kube-api-access-pfm4r\") pod \"maas-api-e2e-ait-f954644c-7646d9f4b-j2q89\" (UID: \"23f022d5-fa7e-46ce-aab7-d2f4008a2ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f954644c-7646d9f4b-j2q89" Jul 23 18:14:04.015472 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:04.015428 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23f022d5-fa7e-46ce-aab7-d2f4008a2ef1-maas-api-tls\") pod \"maas-api-e2e-ait-f954644c-7646d9f4b-j2q89\" (UID: \"23f022d5-fa7e-46ce-aab7-d2f4008a2ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f954644c-7646d9f4b-j2q89" Jul 23 18:14:04.017986 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:04.017955 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23f022d5-fa7e-46ce-aab7-d2f4008a2ef1-maas-api-tls\") pod \"maas-api-e2e-ait-f954644c-7646d9f4b-j2q89\" (UID: \"23f022d5-fa7e-46ce-aab7-d2f4008a2ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f954644c-7646d9f4b-j2q89" Jul 23 18:14:04.266307 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:04.266205 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f954644c-7646d9f4b-j2q89" Jul 23 18:14:04.412507 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:04.412483 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f954644c-7646d9f4b-j2q89"] Jul 23 18:14:04.414632 ip-10-0-137-82 kubenswrapper[2565]: W0723 18:14:04.414580 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f022d5_fa7e_46ce_aab7_d2f4008a2ef1.slice/crio-a940b5533f48143febab8bce3298d2ff34d07030a384d127dd46953ee163e244 WatchSource:0}: Error finding container a940b5533f48143febab8bce3298d2ff34d07030a384d127dd46953ee163e244: Status 404 returned error can't find the container with id a940b5533f48143febab8bce3298d2ff34d07030a384d127dd46953ee163e244 Jul 23 18:14:04.792435 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:04.792400 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f954644c-7646d9f4b-j2q89" event={"ID":"23f022d5-fa7e-46ce-aab7-d2f4008a2ef1","Type":"ContainerStarted","Data":"a940b5533f48143febab8bce3298d2ff34d07030a384d127dd46953ee163e244"} Jul 23 18:14:05.798936 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:05.798901 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f954644c-7646d9f4b-j2q89" event={"ID":"23f022d5-fa7e-46ce-aab7-d2f4008a2ef1","Type":"ContainerStarted","Data":"6c7d0b5561b3867c303a8b5fa666747bd748f0b4e730fb2beed5cbd54d56d97a"} Jul 23 18:14:05.799319 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:05.799041 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f954644c-7646d9f4b-j2q89" Jul 23 18:14:05.818018 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:05.817972 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f954644c-7646d9f4b-j2q89" podStartSLOduration=2.422389635 podStartE2EDuration="2.817956431s" totalImagesPullingTime="395.566796ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:14:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:14:05.816734224 +0000 UTC m=+1789.895037575" watchObservedRunningTime="2026-07-23 18:14:05.817956431 +0000 UTC m=+1789.896259784" Jul 23 18:14:11.808750 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:11.808667 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f954644c-7646d9f4b-j2q89" Jul 23 18:14:16.457181 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:16.457154 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-m7k4j_1cca3236-5c90-4abb-9b9b-39e40b1903c3/console-operator/1.log" Jul 23 18:14:16.462721 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:16.462698 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-m7k4j_1cca3236-5c90-4abb-9b9b-39e40b1903c3/console-operator/1.log" Jul 23 18:14:28.858342 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:28.858303 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f954644c-8nk9v"] Jul 23 18:14:28.861799 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:28.861781 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f954644c-8nk9v" Jul 23 18:14:28.864988 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:28.864969 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5lmm\"" Jul 23 18:14:28.872198 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:28.872172 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f954644c-8nk9v"] Jul 23 18:14:29.047356 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:29.047318 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e97541-e2e7-4083-8c0e-ed816f379aa5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f954644c-8nk9v\" (UID: \"67e97541-e2e7-4083-8c0e-ed816f379aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f954644c-8nk9v" Jul 23 18:14:29.148282 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:29.148192 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e97541-e2e7-4083-8c0e-ed816f379aa5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f954644c-8nk9v\" (UID: \"67e97541-e2e7-4083-8c0e-ed816f379aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f954644c-8nk9v" Jul 23 18:14:29.148786 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:29.148768 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e97541-e2e7-4083-8c0e-ed816f379aa5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f954644c-8nk9v\" (UID: \"67e97541-e2e7-4083-8c0e-ed816f379aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f954644c-8nk9v" Jul 23 18:14:29.172633 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:29.172587 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f954644c-8nk9v" Jul 23 18:14:29.308079 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:29.308052 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f954644c-8nk9v"] Jul 23 18:14:29.310061 ip-10-0-137-82 kubenswrapper[2565]: W0723 18:14:29.310031 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e97541_e2e7_4083_8c0e_ed816f379aa5.slice/crio-722ffc79f2eed3c11df5f4772c3b5bbe65839a76289e4b516fea8c4e712c278c WatchSource:0}: Error finding container 722ffc79f2eed3c11df5f4772c3b5bbe65839a76289e4b516fea8c4e712c278c: Status 404 returned error can't find the container with id 722ffc79f2eed3c11df5f4772c3b5bbe65839a76289e4b516fea8c4e712c278c Jul 23 18:14:29.916694 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:29.916657 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f954644c-8nk9v" event={"ID":"67e97541-e2e7-4083-8c0e-ed816f379aa5","Type":"ContainerStarted","Data":"722ffc79f2eed3c11df5f4772c3b5bbe65839a76289e4b516fea8c4e712c278c"} Jul 23 18:14:31.928697 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:31.928659 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f954644c-8nk9v" event={"ID":"67e97541-e2e7-4083-8c0e-ed816f379aa5","Type":"ContainerStarted","Data":"b60f6dca865d3b5cf4ba6bd3cea14db15c436a6a5801b5022bc8b0a083c7690d"} Jul 23 18:14:31.947848 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:31.947799 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f954644c-8nk9v" podStartSLOduration=1.719849569 podStartE2EDuration="3.947785198s" totalImagesPullingTime="2.227935629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:14:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:14:31.946044839 +0000 UTC m=+1816.024348195" watchObservedRunningTime="2026-07-23 18:14:31.947785198 +0000 UTC m=+1816.026088551" Jul 23 18:14:32.934733 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:32.934695 2565 generic.go:356] "Generic (PLEG): container finished" podID="67e97541-e2e7-4083-8c0e-ed816f379aa5" containerID="b60f6dca865d3b5cf4ba6bd3cea14db15c436a6a5801b5022bc8b0a083c7690d" exitCode=0 Jul 23 18:14:32.935119 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:32.934783 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f954644c-8nk9v" event={"ID":"67e97541-e2e7-4083-8c0e-ed816f379aa5","Type":"ContainerDied","Data":"b60f6dca865d3b5cf4ba6bd3cea14db15c436a6a5801b5022bc8b0a083c7690d"} Jul 23 18:14:34.075429 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:34.075397 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f954644c-8nk9v" Jul 23 18:14:34.093922 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:34.093890 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e97541-e2e7-4083-8c0e-ed816f379aa5-maas-api-service-ca\") pod \"67e97541-e2e7-4083-8c0e-ed816f379aa5\" (UID: \"67e97541-e2e7-4083-8c0e-ed816f379aa5\") " Jul 23 18:14:34.094460 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:34.094282 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e97541-e2e7-4083-8c0e-ed816f379aa5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "67e97541-e2e7-4083-8c0e-ed816f379aa5" (UID: "67e97541-e2e7-4083-8c0e-ed816f379aa5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:14:34.195340 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:34.195255 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e97541-e2e7-4083-8c0e-ed816f379aa5-maas-api-service-ca\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:14:34.945274 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:34.945245 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f954644c-8nk9v" Jul 23 18:14:34.945274 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:34.945269 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f954644c-8nk9v" event={"ID":"67e97541-e2e7-4083-8c0e-ed816f379aa5","Type":"ContainerDied","Data":"722ffc79f2eed3c11df5f4772c3b5bbe65839a76289e4b516fea8c4e712c278c"} Jul 23 18:14:34.945481 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:34.945309 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="722ffc79f2eed3c11df5f4772c3b5bbe65839a76289e4b516fea8c4e712c278c" Jul 23 18:14:39.453310 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:39.453265 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f954644c-7646d9f4b-j2q89"] Jul 23 18:14:39.453768 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:39.453549 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f954644c-7646d9f4b-j2q89" podUID="23f022d5-fa7e-46ce-aab7-d2f4008a2ef1" containerName="maas-api" containerID="cri-o://6c7d0b5561b3867c303a8b5fa666747bd748f0b4e730fb2beed5cbd54d56d97a" gracePeriod=30 Jul 23 18:14:39.722044 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:39.722017 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f954644c-7646d9f4b-j2q89" Jul 23 18:14:39.737913 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:39.737886 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23f022d5-fa7e-46ce-aab7-d2f4008a2ef1-maas-api-tls\") pod \"23f022d5-fa7e-46ce-aab7-d2f4008a2ef1\" (UID: \"23f022d5-fa7e-46ce-aab7-d2f4008a2ef1\") " Jul 23 18:14:39.738062 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:39.737947 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23f022d5-fa7e-46ce-aab7-d2f4008a2ef1-openshift-service-ca\") pod \"23f022d5-fa7e-46ce-aab7-d2f4008a2ef1\" (UID: \"23f022d5-fa7e-46ce-aab7-d2f4008a2ef1\") " Jul 23 18:14:39.738062 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:39.738039 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfm4r\" (UniqueName: \"kubernetes.io/projected/23f022d5-fa7e-46ce-aab7-d2f4008a2ef1-kube-api-access-pfm4r\") pod \"23f022d5-fa7e-46ce-aab7-d2f4008a2ef1\" (UID: \"23f022d5-fa7e-46ce-aab7-d2f4008a2ef1\") " Jul 23 18:14:39.738341 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:39.738262 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23f022d5-fa7e-46ce-aab7-d2f4008a2ef1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "23f022d5-fa7e-46ce-aab7-d2f4008a2ef1" (UID: "23f022d5-fa7e-46ce-aab7-d2f4008a2ef1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:14:39.740526 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:39.740498 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23f022d5-fa7e-46ce-aab7-d2f4008a2ef1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "23f022d5-fa7e-46ce-aab7-d2f4008a2ef1" (UID: "23f022d5-fa7e-46ce-aab7-d2f4008a2ef1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:14:39.740718 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:39.740685 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f022d5-fa7e-46ce-aab7-d2f4008a2ef1-kube-api-access-pfm4r" (OuterVolumeSpecName: "kube-api-access-pfm4r") pod "23f022d5-fa7e-46ce-aab7-d2f4008a2ef1" (UID: "23f022d5-fa7e-46ce-aab7-d2f4008a2ef1"). InnerVolumeSpecName "kube-api-access-pfm4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:14:39.839064 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:39.839025 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23f022d5-fa7e-46ce-aab7-d2f4008a2ef1-openshift-service-ca\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:14:39.839231 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:39.839079 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfm4r\" (UniqueName: \"kubernetes.io/projected/23f022d5-fa7e-46ce-aab7-d2f4008a2ef1-kube-api-access-pfm4r\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:14:39.839231 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:39.839090 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23f022d5-fa7e-46ce-aab7-d2f4008a2ef1-maas-api-tls\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:14:39.977043 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:39.976952 2565 generic.go:356] "Generic (PLEG): container finished" podID="23f022d5-fa7e-46ce-aab7-d2f4008a2ef1" containerID="6c7d0b5561b3867c303a8b5fa666747bd748f0b4e730fb2beed5cbd54d56d97a" exitCode=0 Jul 23 18:14:39.977043 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:39.977037 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f954644c-7646d9f4b-j2q89" Jul 23 18:14:39.977265 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:39.977037 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f954644c-7646d9f4b-j2q89" event={"ID":"23f022d5-fa7e-46ce-aab7-d2f4008a2ef1","Type":"ContainerDied","Data":"6c7d0b5561b3867c303a8b5fa666747bd748f0b4e730fb2beed5cbd54d56d97a"} Jul 23 18:14:39.977265 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:39.977164 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f954644c-7646d9f4b-j2q89" event={"ID":"23f022d5-fa7e-46ce-aab7-d2f4008a2ef1","Type":"ContainerDied","Data":"a940b5533f48143febab8bce3298d2ff34d07030a384d127dd46953ee163e244"} Jul 23 18:14:39.977265 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:39.977203 2565 scope.go:117] "RemoveContainer" containerID="6c7d0b5561b3867c303a8b5fa666747bd748f0b4e730fb2beed5cbd54d56d97a" Jul 23 18:14:39.987398 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:39.987378 2565 scope.go:117] "RemoveContainer" containerID="6c7d0b5561b3867c303a8b5fa666747bd748f0b4e730fb2beed5cbd54d56d97a" Jul 23 18:14:39.987710 ip-10-0-137-82 kubenswrapper[2565]: E0723 18:14:39.987687 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c7d0b5561b3867c303a8b5fa666747bd748f0b4e730fb2beed5cbd54d56d97a\": container with ID starting with 6c7d0b5561b3867c303a8b5fa666747bd748f0b4e730fb2beed5cbd54d56d97a not found: ID does not exist" containerID="6c7d0b5561b3867c303a8b5fa666747bd748f0b4e730fb2beed5cbd54d56d97a" Jul 23 18:14:39.987785 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:39.987717 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c7d0b5561b3867c303a8b5fa666747bd748f0b4e730fb2beed5cbd54d56d97a"} err="failed to get container status \"6c7d0b5561b3867c303a8b5fa666747bd748f0b4e730fb2beed5cbd54d56d97a\": rpc error: code = NotFound desc = could not find container \"6c7d0b5561b3867c303a8b5fa666747bd748f0b4e730fb2beed5cbd54d56d97a\": container with ID starting with 6c7d0b5561b3867c303a8b5fa666747bd748f0b4e730fb2beed5cbd54d56d97a not found: ID does not exist" Jul 23 18:14:40.000407 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:40.000382 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f954644c-7646d9f4b-j2q89"] Jul 23 18:14:40.006117 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:40.006097 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f954644c-7646d9f4b-j2q89"] Jul 23 18:14:40.405543 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:40.405506 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23f022d5-fa7e-46ce-aab7-d2f4008a2ef1" path="/var/lib/kubelet/pods/23f022d5-fa7e-46ce-aab7-d2f4008a2ef1/volumes" Jul 23 18:14:43.952534 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:43.952496 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f954644c-8nk9v"] Jul 23 18:14:43.954187 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:43.954161 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f954644c-8nk9v"] Jul 23 18:14:44.405320 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:44.405285 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67e97541-e2e7-4083-8c0e-ed816f379aa5" path="/var/lib/kubelet/pods/67e97541-e2e7-4083-8c0e-ed816f379aa5/volumes" Jul 23 18:14:48.337807 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:48.337770 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s"] Jul 23 18:14:48.338258 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:48.338235 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23f022d5-fa7e-46ce-aab7-d2f4008a2ef1" containerName="maas-api" Jul 23 18:14:48.338258 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:48.338251 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="23f022d5-fa7e-46ce-aab7-d2f4008a2ef1" containerName="maas-api" Jul 23 18:14:48.338367 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:48.338260 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67e97541-e2e7-4083-8c0e-ed816f379aa5" containerName="revoke-keys" Jul 23 18:14:48.338367 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:48.338267 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="67e97541-e2e7-4083-8c0e-ed816f379aa5" containerName="revoke-keys" Jul 23 18:14:48.338367 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:48.338354 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="67e97541-e2e7-4083-8c0e-ed816f379aa5" containerName="revoke-keys" Jul 23 18:14:48.338367 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:48.338362 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="23f022d5-fa7e-46ce-aab7-d2f4008a2ef1" containerName="maas-api" Jul 23 18:14:48.342652 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:48.342631 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s" Jul 23 18:14:48.345028 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:48.345005 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-f5c76fce-serving-cert\"" Jul 23 18:14:48.355429 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:48.355393 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s"] Jul 23 18:14:48.415413 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:48.415376 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tff85\" (UniqueName: \"kubernetes.io/projected/2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7-kube-api-access-tff85\") pod \"maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s\" (UID: \"2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s" Jul 23 18:14:48.415585 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:48.415426 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7-maas-api-tls\") pod \"maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s\" (UID: \"2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s" Jul 23 18:14:48.415585 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:48.415549 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7-openshift-service-ca\") pod \"maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s\" (UID: \"2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s" Jul 23 18:14:48.516422 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:48.516379 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7-openshift-service-ca\") pod \"maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s\" (UID: \"2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s" Jul 23 18:14:48.516422 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:48.516422 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tff85\" (UniqueName: \"kubernetes.io/projected/2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7-kube-api-access-tff85\") pod \"maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s\" (UID: \"2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s" Jul 23 18:14:48.516715 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:48.516469 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7-maas-api-tls\") pod \"maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s\" (UID: \"2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s" Jul 23 18:14:48.516715 ip-10-0-137-82 kubenswrapper[2565]: E0723 18:14:48.516574 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-f5c76fce-serving-cert: secret "maas-api-e2e-derive-f5c76fce-serving-cert" not found Jul 23 18:14:48.516715 ip-10-0-137-82 kubenswrapper[2565]: E0723 18:14:48.516680 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7-maas-api-tls podName:2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7 nodeName:}" failed. No retries permitted until 2026-07-23 18:14:49.01665546 +0000 UTC m=+1833.094958814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7-maas-api-tls") pod "maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s" (UID: "2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7") : secret "maas-api-e2e-derive-f5c76fce-serving-cert" not found Jul 23 18:14:48.517195 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:48.517173 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7-openshift-service-ca\") pod \"maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s\" (UID: \"2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s" Jul 23 18:14:48.530194 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:48.530164 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tff85\" (UniqueName: \"kubernetes.io/projected/2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7-kube-api-access-tff85\") pod \"maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s\" (UID: \"2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s" Jul 23 18:14:49.021141 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:49.021030 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7-maas-api-tls\") pod \"maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s\" (UID: \"2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s" Jul 23 18:14:49.023805 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:49.023770 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7-maas-api-tls\") pod \"maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s\" (UID: \"2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s" Jul 23 18:14:49.253670 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:49.253629 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s" Jul 23 18:14:49.397250 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:49.397222 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s"] Jul 23 18:14:49.399781 ip-10-0-137-82 kubenswrapper[2565]: W0723 18:14:49.399744 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d98ece7_f1c5_4d4c_89dd_6a7c0ca8b3f7.slice/crio-2bf0ac96c42393509624115dbc84211fdb9b4137f3bc2c8066a9dfb84da2ab4a WatchSource:0}: Error finding container 2bf0ac96c42393509624115dbc84211fdb9b4137f3bc2c8066a9dfb84da2ab4a: Status 404 returned error can't find the container with id 2bf0ac96c42393509624115dbc84211fdb9b4137f3bc2c8066a9dfb84da2ab4a Jul 23 18:14:50.027803 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:50.027765 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s" event={"ID":"2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7","Type":"ContainerStarted","Data":"342b33ed59c094ba307641c748407aed0eb3bee6288a8f22188e75367a136319"} Jul 23 18:14:50.027969 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:50.027847 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s" event={"ID":"2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7","Type":"ContainerStarted","Data":"2bf0ac96c42393509624115dbc84211fdb9b4137f3bc2c8066a9dfb84da2ab4a"} Jul 23 18:14:50.027969 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:50.027881 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s" Jul 23 18:14:50.046608 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:50.046536 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s" podStartSLOduration=1.617617113 podStartE2EDuration="2.046519215s" totalImagesPullingTime="428.902102ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:14:50.045589345 +0000 UTC m=+1834.123892717" watchObservedRunningTime="2026-07-23 18:14:50.046519215 +0000 UTC m=+1834.124822569" Jul 23 18:14:56.038794 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:14:56.038768 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s" Jul 23 18:15:00.133033 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:00.132989 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f5c76fce-29747175-45j2r"] Jul 23 18:15:00.137132 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:00.137111 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747175-gr796"] Jul 23 18:15:00.137253 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:00.137236 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f5c76fce-29747175-45j2r" Jul 23 18:15:00.140136 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:00.140113 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5lmm\"" Jul 23 18:15:00.140498 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:00.140481 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747175-gr796" Jul 23 18:15:00.144028 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:00.144001 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f5c76fce-29747175-45j2r"] Jul 23 18:15:00.147499 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:00.147456 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747175-gr796"] Jul 23 18:15:00.151237 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:00.151206 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f5c76fce-29747175-45j2r" Jul 23 18:15:00.159575 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:00.159554 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747175-gr796" Jul 23 18:15:00.329383 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:00.329348 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f5c76fce-29747175-45j2r"] Jul 23 18:15:00.334103 ip-10-0-137-82 kubenswrapper[2565]: W0723 18:15:00.334066 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69430508_a190_49a0_8248_3568c0ec0aa3.slice/crio-03268dfba625339a4cee4f80bdc6f8212182f1e60d67e96b96b5b58e94d88a96 WatchSource:0}: Error finding container 03268dfba625339a4cee4f80bdc6f8212182f1e60d67e96b96b5b58e94d88a96: Status 404 returned error can't find the container with id 03268dfba625339a4cee4f80bdc6f8212182f1e60d67e96b96b5b58e94d88a96 Jul 23 18:15:00.360728 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:00.360700 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747175-gr796"] Jul 23 18:15:01.081162 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:01.081124 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f5c76fce-29747175-45j2r" event={"ID":"69430508-a190-49a0-8248-3568c0ec0aa3","Type":"ContainerStarted","Data":"0ff2d94749749ed6e70cf311bf163efb0e94ccee2e551a7b634cc03f9a6ac80a"} Jul 23 18:15:01.081351 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:01.081200 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f5c76fce-29747175-45j2r" event={"ID":"69430508-a190-49a0-8248-3568c0ec0aa3","Type":"ContainerStarted","Data":"03268dfba625339a4cee4f80bdc6f8212182f1e60d67e96b96b5b58e94d88a96"} Jul 23 18:15:01.082655 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:01.082625 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747175-gr796" event={"ID":"a580b0b0-f010-4c4c-aad2-43590ad632ac","Type":"ContainerStarted","Data":"4f1703af47602a035174c1572962eb3ece645940a1e40ac32c12d3dac629dcd8"} Jul 23 18:15:01.082756 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:01.082681 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747175-gr796" event={"ID":"a580b0b0-f010-4c4c-aad2-43590ad632ac","Type":"ContainerStarted","Data":"a91f6b18ead51d24f4f95b61c5cd8af4a12fd0722be663601a9a8996e66b3349"} Jul 23 18:15:01.099863 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:01.099795 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f5c76fce-29747175-45j2r" podStartSLOduration=1.099767258 podStartE2EDuration="1.099767258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:15:01.097074659 +0000 UTC m=+1845.175378004" watchObservedRunningTime="2026-07-23 18:15:01.099767258 +0000 UTC m=+1845.178070615" Jul 23 18:15:01.116646 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:01.116544 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747175-gr796" podStartSLOduration=1.116525974 podStartE2EDuration="1.116525974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:15:01.116469398 +0000 UTC m=+1845.194772755" watchObservedRunningTime="2026-07-23 18:15:01.116525974 +0000 UTC m=+1845.194829329" Jul 23 18:15:02.091808 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:02.091773 2565 generic.go:356] "Generic (PLEG): container finished" podID="69430508-a190-49a0-8248-3568c0ec0aa3" containerID="0ff2d94749749ed6e70cf311bf163efb0e94ccee2e551a7b634cc03f9a6ac80a" exitCode=0 Jul 23 18:15:02.092238 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:02.091864 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f5c76fce-29747175-45j2r" event={"ID":"69430508-a190-49a0-8248-3568c0ec0aa3","Type":"ContainerDied","Data":"0ff2d94749749ed6e70cf311bf163efb0e94ccee2e551a7b634cc03f9a6ac80a"} Jul 23 18:15:02.093390 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:02.093357 2565 generic.go:356] "Generic (PLEG): container finished" podID="a580b0b0-f010-4c4c-aad2-43590ad632ac" containerID="4f1703af47602a035174c1572962eb3ece645940a1e40ac32c12d3dac629dcd8" exitCode=0 Jul 23 18:15:02.093508 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:02.093439 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747175-gr796" event={"ID":"a580b0b0-f010-4c4c-aad2-43590ad632ac","Type":"ContainerDied","Data":"4f1703af47602a035174c1572962eb3ece645940a1e40ac32c12d3dac629dcd8"} Jul 23 18:15:03.237876 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:03.237851 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747175-gr796" Jul 23 18:15:03.255073 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:03.255047 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f5c76fce-29747175-45j2r" Jul 23 18:15:04.105007 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:04.104981 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747175-gr796" Jul 23 18:15:04.105195 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:04.105022 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747175-gr796" event={"ID":"a580b0b0-f010-4c4c-aad2-43590ad632ac","Type":"ContainerDied","Data":"a91f6b18ead51d24f4f95b61c5cd8af4a12fd0722be663601a9a8996e66b3349"} Jul 23 18:15:04.105195 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:04.105069 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a91f6b18ead51d24f4f95b61c5cd8af4a12fd0722be663601a9a8996e66b3349" Jul 23 18:15:04.106357 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:04.106332 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f5c76fce-29747175-45j2r" event={"ID":"69430508-a190-49a0-8248-3568c0ec0aa3","Type":"ContainerDied","Data":"03268dfba625339a4cee4f80bdc6f8212182f1e60d67e96b96b5b58e94d88a96"} Jul 23 18:15:04.106459 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:04.106371 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03268dfba625339a4cee4f80bdc6f8212182f1e60d67e96b96b5b58e94d88a96" Jul 23 18:15:04.106459 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:04.106339 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f5c76fce-29747175-45j2r" Jul 23 18:15:13.105135 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:13.105085 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5c76fce-965xf"] Jul 23 18:15:13.105614 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:13.105470 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69430508-a190-49a0-8248-3568c0ec0aa3" containerName="cleanup" Jul 23 18:15:13.105614 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:13.105480 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="69430508-a190-49a0-8248-3568c0ec0aa3" containerName="cleanup" Jul 23 18:15:13.105614 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:13.105494 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a580b0b0-f010-4c4c-aad2-43590ad632ac" containerName="cleanup" Jul 23 18:15:13.105614 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:13.105500 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a580b0b0-f010-4c4c-aad2-43590ad632ac" containerName="cleanup" Jul 23 18:15:13.105614 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:13.105564 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="69430508-a190-49a0-8248-3568c0ec0aa3" containerName="cleanup" Jul 23 18:15:13.105614 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:13.105575 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a580b0b0-f010-4c4c-aad2-43590ad632ac" containerName="cleanup" Jul 23 18:15:13.112102 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:13.112080 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5c76fce-965xf" Jul 23 18:15:13.115093 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:13.115068 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5lmm\"" Jul 23 18:15:13.121769 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:13.119906 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5c76fce-965xf"] Jul 23 18:15:13.229810 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:13.229770 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6415a9-6c16-4817-be00-d02ff2afaf93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f5c76fce-965xf\" (UID: \"8f6415a9-6c16-4817-be00-d02ff2afaf93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5c76fce-965xf" Jul 23 18:15:13.330930 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:13.330874 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6415a9-6c16-4817-be00-d02ff2afaf93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f5c76fce-965xf\" (UID: \"8f6415a9-6c16-4817-be00-d02ff2afaf93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5c76fce-965xf" Jul 23 18:15:13.331519 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:13.331496 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6415a9-6c16-4817-be00-d02ff2afaf93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f5c76fce-965xf\" (UID: \"8f6415a9-6c16-4817-be00-d02ff2afaf93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5c76fce-965xf" Jul 23 18:15:13.426436 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:13.426356 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5c76fce-965xf" Jul 23 18:15:13.562515 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:13.562489 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5c76fce-965xf"] Jul 23 18:15:13.564437 ip-10-0-137-82 kubenswrapper[2565]: W0723 18:15:13.564404 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f6415a9_6c16_4817_be00_d02ff2afaf93.slice/crio-541eb4f65024267e8bdb959254497d4d614569bf7f99bbe6c98ac47513b6d4b4 WatchSource:0}: Error finding container 541eb4f65024267e8bdb959254497d4d614569bf7f99bbe6c98ac47513b6d4b4: Status 404 returned error can't find the container with id 541eb4f65024267e8bdb959254497d4d614569bf7f99bbe6c98ac47513b6d4b4 Jul 23 18:15:14.155014 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:14.154976 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5c76fce-965xf" event={"ID":"8f6415a9-6c16-4817-be00-d02ff2afaf93","Type":"ContainerStarted","Data":"7a232e6ab24ee93a0a3a09288f2266495b594b429b59f7cc1aab34b71ee40e22"} Jul 23 18:15:14.155390 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:14.155040 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5c76fce-965xf" event={"ID":"8f6415a9-6c16-4817-be00-d02ff2afaf93","Type":"ContainerStarted","Data":"541eb4f65024267e8bdb959254497d4d614569bf7f99bbe6c98ac47513b6d4b4"} Jul 23 18:15:14.173838 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:14.173793 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5c76fce-965xf" podStartSLOduration=1.17377764 podStartE2EDuration="1.17377764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:15:14.173160484 +0000 UTC m=+1858.251463837" watchObservedRunningTime="2026-07-23 18:15:14.17377764 +0000 UTC m=+1858.252080992" Jul 23 18:15:15.161268 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:15.161234 2565 generic.go:356] "Generic (PLEG): container finished" podID="8f6415a9-6c16-4817-be00-d02ff2afaf93" containerID="7a232e6ab24ee93a0a3a09288f2266495b594b429b59f7cc1aab34b71ee40e22" exitCode=0 Jul 23 18:15:15.161696 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:15.161325 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5c76fce-965xf" event={"ID":"8f6415a9-6c16-4817-be00-d02ff2afaf93","Type":"ContainerDied","Data":"7a232e6ab24ee93a0a3a09288f2266495b594b429b59f7cc1aab34b71ee40e22"} Jul 23 18:15:16.299920 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:16.299894 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5c76fce-965xf" Jul 23 18:15:16.459051 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:16.458969 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6415a9-6c16-4817-be00-d02ff2afaf93-maas-api-service-ca\") pod \"8f6415a9-6c16-4817-be00-d02ff2afaf93\" (UID: \"8f6415a9-6c16-4817-be00-d02ff2afaf93\") " Jul 23 18:15:16.459331 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:16.459307 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f6415a9-6c16-4817-be00-d02ff2afaf93-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8f6415a9-6c16-4817-be00-d02ff2afaf93" (UID: "8f6415a9-6c16-4817-be00-d02ff2afaf93"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:15:16.559672 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:16.559647 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6415a9-6c16-4817-be00-d02ff2afaf93-maas-api-service-ca\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:15:17.172591 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:17.172557 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5c76fce-965xf" Jul 23 18:15:17.172786 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:17.172590 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5c76fce-965xf" event={"ID":"8f6415a9-6c16-4817-be00-d02ff2afaf93","Type":"ContainerDied","Data":"541eb4f65024267e8bdb959254497d4d614569bf7f99bbe6c98ac47513b6d4b4"} Jul 23 18:15:17.172786 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:17.172663 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="541eb4f65024267e8bdb959254497d4d614569bf7f99bbe6c98ac47513b6d4b4" Jul 23 18:15:23.633609 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:23.633565 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s"] Jul 23 18:15:23.633974 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:23.633784 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s" podUID="2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7" containerName="maas-api" containerID="cri-o://342b33ed59c094ba307641c748407aed0eb3bee6288a8f22188e75367a136319" gracePeriod=30 Jul 23 18:15:23.666349 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:23.666314 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f5c76fce-29747175-45j2r"] Jul 23 18:15:23.670571 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:23.670545 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f5c76fce-29747175-45j2r"] Jul 23 18:15:23.888393 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:23.888329 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s" Jul 23 18:15:24.023221 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:24.023191 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7-openshift-service-ca\") pod \"2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7\" (UID: \"2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7\") " Jul 23 18:15:24.023400 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:24.023244 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tff85\" (UniqueName: \"kubernetes.io/projected/2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7-kube-api-access-tff85\") pod \"2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7\" (UID: \"2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7\") " Jul 23 18:15:24.023400 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:24.023289 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7-maas-api-tls\") pod \"2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7\" (UID: \"2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7\") " Jul 23 18:15:24.023586 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:24.023560 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7" (UID: "2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:15:24.025431 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:24.025402 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7" (UID: "2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:15:24.025541 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:24.025477 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7-kube-api-access-tff85" (OuterVolumeSpecName: "kube-api-access-tff85") pod "2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7" (UID: "2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7"). InnerVolumeSpecName "kube-api-access-tff85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:15:24.132469 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:24.132418 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tff85\" (UniqueName: \"kubernetes.io/projected/2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7-kube-api-access-tff85\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:15:24.132676 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:24.132619 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7-maas-api-tls\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:15:24.132676 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:24.132643 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7-openshift-service-ca\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:15:24.213960 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:24.213868 2565 generic.go:356] "Generic (PLEG): container finished" podID="2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7" containerID="342b33ed59c094ba307641c748407aed0eb3bee6288a8f22188e75367a136319" exitCode=0 Jul 23 18:15:24.213960 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:24.213945 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s" Jul 23 18:15:24.214182 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:24.213945 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s" event={"ID":"2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7","Type":"ContainerDied","Data":"342b33ed59c094ba307641c748407aed0eb3bee6288a8f22188e75367a136319"} Jul 23 18:15:24.214182 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:24.214105 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s" event={"ID":"2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7","Type":"ContainerDied","Data":"2bf0ac96c42393509624115dbc84211fdb9b4137f3bc2c8066a9dfb84da2ab4a"} Jul 23 18:15:24.214182 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:24.214132 2565 scope.go:117] "RemoveContainer" containerID="342b33ed59c094ba307641c748407aed0eb3bee6288a8f22188e75367a136319" Jul 23 18:15:24.224086 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:24.224062 2565 scope.go:117] "RemoveContainer" containerID="342b33ed59c094ba307641c748407aed0eb3bee6288a8f22188e75367a136319" Jul 23 18:15:24.224343 ip-10-0-137-82 kubenswrapper[2565]: E0723 18:15:24.224322 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"342b33ed59c094ba307641c748407aed0eb3bee6288a8f22188e75367a136319\": container with ID starting with 342b33ed59c094ba307641c748407aed0eb3bee6288a8f22188e75367a136319 not found: ID does not exist" containerID="342b33ed59c094ba307641c748407aed0eb3bee6288a8f22188e75367a136319" Jul 23 18:15:24.224400 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:24.224352 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"342b33ed59c094ba307641c748407aed0eb3bee6288a8f22188e75367a136319"} err="failed to get container status \"342b33ed59c094ba307641c748407aed0eb3bee6288a8f22188e75367a136319\": rpc error: code = NotFound desc = could not find container \"342b33ed59c094ba307641c748407aed0eb3bee6288a8f22188e75367a136319\": container with ID starting with 342b33ed59c094ba307641c748407aed0eb3bee6288a8f22188e75367a136319 not found: ID does not exist" Jul 23 18:15:24.237044 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:24.237017 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s"] Jul 23 18:15:24.241267 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:24.241238 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f5c76fce-65d6bb57f7-2nl5s"] Jul 23 18:15:24.406123 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:24.406091 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7" path="/var/lib/kubelet/pods/2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7/volumes" Jul 23 18:15:24.406387 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:24.406374 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69430508-a190-49a0-8248-3568c0ec0aa3" path="/var/lib/kubelet/pods/69430508-a190-49a0-8248-3568c0ec0aa3/volumes" Jul 23 18:15:28.224690 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:28.224648 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5c76fce-965xf"] Jul 23 18:15:28.227136 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:28.227111 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5c76fce-965xf"] Jul 23 18:15:28.406331 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:15:28.406294 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f6415a9-6c16-4817-be00-d02ff2afaf93" path="/var/lib/kubelet/pods/8f6415a9-6c16-4817-be00-d02ff2afaf93/volumes" Jul 23 18:17:11.874759 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:11.874675 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz"] Jul 23 18:17:11.877306 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:11.875238 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f6415a9-6c16-4817-be00-d02ff2afaf93" containerName="revoke-keys" Jul 23 18:17:11.877306 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:11.875255 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f6415a9-6c16-4817-be00-d02ff2afaf93" containerName="revoke-keys" Jul 23 18:17:11.877306 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:11.875276 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7" containerName="maas-api" Jul 23 18:17:11.877306 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:11.875285 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7" containerName="maas-api" Jul 23 18:17:11.877306 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:11.875366 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f6415a9-6c16-4817-be00-d02ff2afaf93" containerName="revoke-keys" Jul 23 18:17:11.877306 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:11.875380 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d98ece7-f1c5-4d4c-89dd-6a7c0ca8b3f7" containerName="maas-api" Jul 23 18:17:11.878777 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:11.878754 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz" Jul 23 18:17:11.881483 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:11.881458 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-99f42efd-serving-cert\"" Jul 23 18:17:11.888078 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:11.888051 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz"] Jul 23 18:17:11.975520 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:11.975485 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5814074d-d583-4ed3-9cbb-5546550e324b-openshift-service-ca\") pod \"maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz\" (UID: \"5814074d-d583-4ed3-9cbb-5546550e324b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz" Jul 23 18:17:11.975739 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:11.975572 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5814074d-d583-4ed3-9cbb-5546550e324b-maas-api-tls\") pod \"maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz\" (UID: \"5814074d-d583-4ed3-9cbb-5546550e324b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz" Jul 23 18:17:11.975739 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:11.975623 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m4dc\" (UniqueName: \"kubernetes.io/projected/5814074d-d583-4ed3-9cbb-5546550e324b-kube-api-access-5m4dc\") pod \"maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz\" (UID: \"5814074d-d583-4ed3-9cbb-5546550e324b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz" Jul 23 18:17:12.076251 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:12.076213 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5814074d-d583-4ed3-9cbb-5546550e324b-openshift-service-ca\") pod \"maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz\" (UID: \"5814074d-d583-4ed3-9cbb-5546550e324b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz" Jul 23 18:17:12.076413 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:12.076359 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5814074d-d583-4ed3-9cbb-5546550e324b-maas-api-tls\") pod \"maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz\" (UID: \"5814074d-d583-4ed3-9cbb-5546550e324b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz" Jul 23 18:17:12.076413 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:12.076380 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m4dc\" (UniqueName: \"kubernetes.io/projected/5814074d-d583-4ed3-9cbb-5546550e324b-kube-api-access-5m4dc\") pod \"maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz\" (UID: \"5814074d-d583-4ed3-9cbb-5546550e324b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz" Jul 23 18:17:12.076973 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:12.076944 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5814074d-d583-4ed3-9cbb-5546550e324b-openshift-service-ca\") pod \"maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz\" (UID: \"5814074d-d583-4ed3-9cbb-5546550e324b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz" Jul 23 18:17:12.078743 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:12.078720 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5814074d-d583-4ed3-9cbb-5546550e324b-maas-api-tls\") pod \"maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz\" (UID: \"5814074d-d583-4ed3-9cbb-5546550e324b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz" Jul 23 18:17:12.084585 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:12.084559 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m4dc\" (UniqueName: \"kubernetes.io/projected/5814074d-d583-4ed3-9cbb-5546550e324b-kube-api-access-5m4dc\") pod \"maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz\" (UID: \"5814074d-d583-4ed3-9cbb-5546550e324b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz" Jul 23 18:17:12.192248 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:12.192170 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz" Jul 23 18:17:12.546084 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:12.546058 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz"] Jul 23 18:17:12.547931 ip-10-0-137-82 kubenswrapper[2565]: W0723 18:17:12.547898 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5814074d_d583_4ed3_9cbb_5546550e324b.slice/crio-92ea38670d34f97ef1c8925a2389d6fad312a782106223d30709854bb0b2c9c6 WatchSource:0}: Error finding container 92ea38670d34f97ef1c8925a2389d6fad312a782106223d30709854bb0b2c9c6: Status 404 returned error can't find the container with id 92ea38670d34f97ef1c8925a2389d6fad312a782106223d30709854bb0b2c9c6 Jul 23 18:17:12.760530 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:12.760493 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz" event={"ID":"5814074d-d583-4ed3-9cbb-5546550e324b","Type":"ContainerStarted","Data":"92ea38670d34f97ef1c8925a2389d6fad312a782106223d30709854bb0b2c9c6"} Jul 23 18:17:13.766784 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:13.766747 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz" event={"ID":"5814074d-d583-4ed3-9cbb-5546550e324b","Type":"ContainerStarted","Data":"b6df19b4ba45af27281cebc08055ca5cf935d521e008f5defdf2b5b30d03a026"} Jul 23 18:17:13.766784 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:13.766805 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz" Jul 23 18:17:13.785702 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:13.785653 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz" podStartSLOduration=2.241521833 podStartE2EDuration="2.785635159s" totalImagesPullingTime="544.113326ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:17:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:13.784068527 +0000 UTC m=+1977.862371880" watchObservedRunningTime="2026-07-23 18:17:13.785635159 +0000 UTC m=+1977.863938512" Jul 23 18:17:19.776936 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:19.776908 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz" Jul 23 18:17:23.508837 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:23.506055 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99f42efd-lc6tj"] Jul 23 18:17:23.511034 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:23.511003 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99f42efd-lc6tj" Jul 23 18:17:23.513842 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:23.513804 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5lmm\"" Jul 23 18:17:23.515064 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:23.515037 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99f42efd-lc6tj"] Jul 23 18:17:23.683689 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:23.683654 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/172e0927-56f0-4484-8d53-61735dd2da62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-99f42efd-lc6tj\" (UID: \"172e0927-56f0-4484-8d53-61735dd2da62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99f42efd-lc6tj" Jul 23 18:17:23.784641 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:23.784585 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/172e0927-56f0-4484-8d53-61735dd2da62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-99f42efd-lc6tj\" (UID: \"172e0927-56f0-4484-8d53-61735dd2da62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99f42efd-lc6tj" Jul 23 18:17:23.785212 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:23.785194 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/172e0927-56f0-4484-8d53-61735dd2da62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-99f42efd-lc6tj\" (UID: \"172e0927-56f0-4484-8d53-61735dd2da62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99f42efd-lc6tj" Jul 23 18:17:23.825125 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:23.825099 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99f42efd-lc6tj" Jul 23 18:17:23.959200 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:23.959173 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99f42efd-lc6tj"] Jul 23 18:17:23.961309 ip-10-0-137-82 kubenswrapper[2565]: W0723 18:17:23.961284 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod172e0927_56f0_4484_8d53_61735dd2da62.slice/crio-9e15a0a5e49c179959c7cca1dd797de084273bf89bad18e291192b35cb50c55b WatchSource:0}: Error finding container 9e15a0a5e49c179959c7cca1dd797de084273bf89bad18e291192b35cb50c55b: Status 404 returned error can't find the container with id 9e15a0a5e49c179959c7cca1dd797de084273bf89bad18e291192b35cb50c55b Jul 23 18:17:24.824933 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:24.824898 2565 generic.go:356] "Generic (PLEG): container finished" podID="172e0927-56f0-4484-8d53-61735dd2da62" containerID="4de6ad418321750e51b204289ab35680e818d0fcb1da7ba94c232dc68d7f717f" exitCode=0 Jul 23 18:17:24.825374 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:24.824989 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99f42efd-lc6tj" event={"ID":"172e0927-56f0-4484-8d53-61735dd2da62","Type":"ContainerDied","Data":"4de6ad418321750e51b204289ab35680e818d0fcb1da7ba94c232dc68d7f717f"} Jul 23 18:17:24.825374 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:24.825059 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99f42efd-lc6tj" event={"ID":"172e0927-56f0-4484-8d53-61735dd2da62","Type":"ContainerStarted","Data":"9e15a0a5e49c179959c7cca1dd797de084273bf89bad18e291192b35cb50c55b"} Jul 23 18:17:25.966324 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:25.966296 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99f42efd-lc6tj" Jul 23 18:17:26.105311 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:26.105231 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/172e0927-56f0-4484-8d53-61735dd2da62-maas-api-service-ca\") pod \"172e0927-56f0-4484-8d53-61735dd2da62\" (UID: \"172e0927-56f0-4484-8d53-61735dd2da62\") " Jul 23 18:17:26.105583 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:26.105558 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/172e0927-56f0-4484-8d53-61735dd2da62-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "172e0927-56f0-4484-8d53-61735dd2da62" (UID: "172e0927-56f0-4484-8d53-61735dd2da62"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:17:26.206359 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:26.206315 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/172e0927-56f0-4484-8d53-61735dd2da62-maas-api-service-ca\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:17:26.838223 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:26.838196 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99f42efd-lc6tj" Jul 23 18:17:26.838223 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:26.838214 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99f42efd-lc6tj" event={"ID":"172e0927-56f0-4484-8d53-61735dd2da62","Type":"ContainerDied","Data":"9e15a0a5e49c179959c7cca1dd797de084273bf89bad18e291192b35cb50c55b"} Jul 23 18:17:26.838481 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:26.838261 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e15a0a5e49c179959c7cca1dd797de084273bf89bad18e291192b35cb50c55b" Jul 23 18:17:34.052186 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:34.052128 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz"] Jul 23 18:17:34.052623 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:34.052363 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz" podUID="5814074d-d583-4ed3-9cbb-5546550e324b" containerName="maas-api" containerID="cri-o://b6df19b4ba45af27281cebc08055ca5cf935d521e008f5defdf2b5b30d03a026" gracePeriod=30 Jul 23 18:17:34.306950 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:34.306887 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz" Jul 23 18:17:34.377924 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:34.377881 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5814074d-d583-4ed3-9cbb-5546550e324b-openshift-service-ca\") pod \"5814074d-d583-4ed3-9cbb-5546550e324b\" (UID: \"5814074d-d583-4ed3-9cbb-5546550e324b\") " Jul 23 18:17:34.378123 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:34.377959 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5m4dc\" (UniqueName: \"kubernetes.io/projected/5814074d-d583-4ed3-9cbb-5546550e324b-kube-api-access-5m4dc\") pod \"5814074d-d583-4ed3-9cbb-5546550e324b\" (UID: \"5814074d-d583-4ed3-9cbb-5546550e324b\") " Jul 23 18:17:34.378123 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:34.378055 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5814074d-d583-4ed3-9cbb-5546550e324b-maas-api-tls\") pod \"5814074d-d583-4ed3-9cbb-5546550e324b\" (UID: \"5814074d-d583-4ed3-9cbb-5546550e324b\") " Jul 23 18:17:34.378311 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:34.378281 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5814074d-d583-4ed3-9cbb-5546550e324b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5814074d-d583-4ed3-9cbb-5546550e324b" (UID: "5814074d-d583-4ed3-9cbb-5546550e324b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:17:34.380151 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:34.380118 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5814074d-d583-4ed3-9cbb-5546550e324b-kube-api-access-5m4dc" (OuterVolumeSpecName: "kube-api-access-5m4dc") pod "5814074d-d583-4ed3-9cbb-5546550e324b" (UID: "5814074d-d583-4ed3-9cbb-5546550e324b"). InnerVolumeSpecName "kube-api-access-5m4dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:17:34.380258 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:34.380149 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5814074d-d583-4ed3-9cbb-5546550e324b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5814074d-d583-4ed3-9cbb-5546550e324b" (UID: "5814074d-d583-4ed3-9cbb-5546550e324b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:17:34.479539 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:34.479497 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m4dc\" (UniqueName: \"kubernetes.io/projected/5814074d-d583-4ed3-9cbb-5546550e324b-kube-api-access-5m4dc\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:17:34.479539 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:34.479540 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5814074d-d583-4ed3-9cbb-5546550e324b-maas-api-tls\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:17:34.479759 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:34.479555 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5814074d-d583-4ed3-9cbb-5546550e324b-openshift-service-ca\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:17:34.878901 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:34.878861 2565 generic.go:356] "Generic (PLEG): container finished" podID="5814074d-d583-4ed3-9cbb-5546550e324b" containerID="b6df19b4ba45af27281cebc08055ca5cf935d521e008f5defdf2b5b30d03a026" exitCode=0 Jul 23 18:17:34.878901 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:34.878901 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz" Jul 23 18:17:34.879153 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:34.878933 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz" event={"ID":"5814074d-d583-4ed3-9cbb-5546550e324b","Type":"ContainerDied","Data":"b6df19b4ba45af27281cebc08055ca5cf935d521e008f5defdf2b5b30d03a026"} Jul 23 18:17:34.879153 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:34.878979 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz" event={"ID":"5814074d-d583-4ed3-9cbb-5546550e324b","Type":"ContainerDied","Data":"92ea38670d34f97ef1c8925a2389d6fad312a782106223d30709854bb0b2c9c6"} Jul 23 18:17:34.879153 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:34.879008 2565 scope.go:117] "RemoveContainer" containerID="b6df19b4ba45af27281cebc08055ca5cf935d521e008f5defdf2b5b30d03a026" Jul 23 18:17:34.887904 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:34.887858 2565 scope.go:117] "RemoveContainer" containerID="b6df19b4ba45af27281cebc08055ca5cf935d521e008f5defdf2b5b30d03a026" Jul 23 18:17:34.888182 ip-10-0-137-82 kubenswrapper[2565]: E0723 18:17:34.888157 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6df19b4ba45af27281cebc08055ca5cf935d521e008f5defdf2b5b30d03a026\": container with ID starting with b6df19b4ba45af27281cebc08055ca5cf935d521e008f5defdf2b5b30d03a026 not found: ID does not exist" containerID="b6df19b4ba45af27281cebc08055ca5cf935d521e008f5defdf2b5b30d03a026" Jul 23 18:17:34.888251 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:34.888190 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6df19b4ba45af27281cebc08055ca5cf935d521e008f5defdf2b5b30d03a026"} err="failed to get container status \"b6df19b4ba45af27281cebc08055ca5cf935d521e008f5defdf2b5b30d03a026\": rpc error: code = NotFound desc = could not find container \"b6df19b4ba45af27281cebc08055ca5cf935d521e008f5defdf2b5b30d03a026\": container with ID starting with b6df19b4ba45af27281cebc08055ca5cf935d521e008f5defdf2b5b30d03a026 not found: ID does not exist" Jul 23 18:17:34.898484 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:34.898451 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz"] Jul 23 18:17:34.904563 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:34.904536 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-99f42efd-864bc6c49f-7dzqz"] Jul 23 18:17:36.405978 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:36.405946 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5814074d-d583-4ed3-9cbb-5546550e324b" path="/var/lib/kubelet/pods/5814074d-d583-4ed3-9cbb-5546550e324b/volumes" Jul 23 18:17:38.593874 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:38.593832 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99f42efd-lc6tj"] Jul 23 18:17:38.596145 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:38.596119 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99f42efd-lc6tj"] Jul 23 18:17:40.405663 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:17:40.405628 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="172e0927-56f0-4484-8d53-61735dd2da62" path="/var/lib/kubelet/pods/172e0927-56f0-4484-8d53-61735dd2da62/volumes" Jul 23 18:18:07.104829 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:07.104782 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c"] Jul 23 18:18:07.105421 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:07.105207 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="172e0927-56f0-4484-8d53-61735dd2da62" containerName="revoke-keys" Jul 23 18:18:07.105421 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:07.105221 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="172e0927-56f0-4484-8d53-61735dd2da62" containerName="revoke-keys" Jul 23 18:18:07.105421 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:07.105231 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5814074d-d583-4ed3-9cbb-5546550e324b" containerName="maas-api" Jul 23 18:18:07.105421 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:07.105238 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5814074d-d583-4ed3-9cbb-5546550e324b" containerName="maas-api" Jul 23 18:18:07.105421 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:07.105300 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="172e0927-56f0-4484-8d53-61735dd2da62" containerName="revoke-keys" Jul 23 18:18:07.105421 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:07.105310 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5814074d-d583-4ed3-9cbb-5546550e324b" containerName="maas-api" Jul 23 18:18:07.108573 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:07.108549 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c" Jul 23 18:18:07.111480 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:07.111458 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-def6e9dd-serving-cert\"" Jul 23 18:18:07.128690 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:07.128650 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c"] Jul 23 18:18:07.277351 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:07.277314 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz7hf\" (UniqueName: \"kubernetes.io/projected/93dda095-70be-49a5-b461-5b2d5c282078-kube-api-access-zz7hf\") pod \"maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c\" (UID: \"93dda095-70be-49a5-b461-5b2d5c282078\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c" Jul 23 18:18:07.277567 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:07.277376 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93dda095-70be-49a5-b461-5b2d5c282078-maas-api-tls\") pod \"maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c\" (UID: \"93dda095-70be-49a5-b461-5b2d5c282078\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c" Jul 23 18:18:07.277567 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:07.277435 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93dda095-70be-49a5-b461-5b2d5c282078-openshift-service-ca\") pod \"maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c\" (UID: \"93dda095-70be-49a5-b461-5b2d5c282078\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c" Jul 23 18:18:07.378070 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:07.377968 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz7hf\" (UniqueName: \"kubernetes.io/projected/93dda095-70be-49a5-b461-5b2d5c282078-kube-api-access-zz7hf\") pod \"maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c\" (UID: \"93dda095-70be-49a5-b461-5b2d5c282078\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c" Jul 23 18:18:07.378070 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:07.378030 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93dda095-70be-49a5-b461-5b2d5c282078-maas-api-tls\") pod \"maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c\" (UID: \"93dda095-70be-49a5-b461-5b2d5c282078\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c" Jul 23 18:18:07.378311 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:07.378080 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93dda095-70be-49a5-b461-5b2d5c282078-openshift-service-ca\") pod \"maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c\" (UID: \"93dda095-70be-49a5-b461-5b2d5c282078\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c" Jul 23 18:18:07.378890 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:07.378862 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93dda095-70be-49a5-b461-5b2d5c282078-openshift-service-ca\") pod \"maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c\" (UID: \"93dda095-70be-49a5-b461-5b2d5c282078\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c" Jul 23 18:18:07.380505 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:07.380475 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93dda095-70be-49a5-b461-5b2d5c282078-maas-api-tls\") pod \"maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c\" (UID: \"93dda095-70be-49a5-b461-5b2d5c282078\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c" Jul 23 18:18:07.390913 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:07.390891 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz7hf\" (UniqueName: \"kubernetes.io/projected/93dda095-70be-49a5-b461-5b2d5c282078-kube-api-access-zz7hf\") pod \"maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c\" (UID: \"93dda095-70be-49a5-b461-5b2d5c282078\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c" Jul 23 18:18:07.419870 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:07.419842 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c" Jul 23 18:18:07.563379 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:07.563346 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c"] Jul 23 18:18:07.564962 ip-10-0-137-82 kubenswrapper[2565]: W0723 18:18:07.564935 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93dda095_70be_49a5_b461_5b2d5c282078.slice/crio-29bf884717420efe0e001f639e18153e880786343ae8cdb185b24efec57e60f0 WatchSource:0}: Error finding container 29bf884717420efe0e001f639e18153e880786343ae8cdb185b24efec57e60f0: Status 404 returned error can't find the container with id 29bf884717420efe0e001f639e18153e880786343ae8cdb185b24efec57e60f0 Jul 23 18:18:07.566278 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:07.566260 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 18:18:08.045288 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:08.045252 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c" event={"ID":"93dda095-70be-49a5-b461-5b2d5c282078","Type":"ContainerStarted","Data":"29bf884717420efe0e001f639e18153e880786343ae8cdb185b24efec57e60f0"} Jul 23 18:18:09.053072 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:09.053033 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c" event={"ID":"93dda095-70be-49a5-b461-5b2d5c282078","Type":"ContainerStarted","Data":"70e1fe422690e3fe4d4bd401b2c688171e3e62479b247dca4d2b805381378242"} Jul 23 18:18:09.053493 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:09.053104 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c" Jul 23 18:18:09.070774 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:09.070722 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c" podStartSLOduration=1.587146001 podStartE2EDuration="2.070705833s" totalImagesPullingTime="483.559832ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:18:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:18:09.068791108 +0000 UTC m=+2033.147094459" watchObservedRunningTime="2026-07-23 18:18:09.070705833 +0000 UTC m=+2033.149009186" Jul 23 18:18:15.064281 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:15.064253 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c" Jul 23 18:18:50.272821 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:50.272740 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-def6e9dd-7hzmq"] Jul 23 18:18:50.276272 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:50.276255 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-def6e9dd-7hzmq" Jul 23 18:18:50.278846 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:50.278827 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5lmm\"" Jul 23 18:18:50.284741 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:50.284716 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-def6e9dd-7hzmq"] Jul 23 18:18:50.338514 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:50.338478 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6610e788-85c7-4a3c-aae5-d91ce068537e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-def6e9dd-7hzmq\" (UID: \"6610e788-85c7-4a3c-aae5-d91ce068537e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-def6e9dd-7hzmq" Jul 23 18:18:50.439574 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:50.439541 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6610e788-85c7-4a3c-aae5-d91ce068537e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-def6e9dd-7hzmq\" (UID: \"6610e788-85c7-4a3c-aae5-d91ce068537e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-def6e9dd-7hzmq" Jul 23 18:18:50.440188 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:50.440167 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6610e788-85c7-4a3c-aae5-d91ce068537e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-def6e9dd-7hzmq\" (UID: \"6610e788-85c7-4a3c-aae5-d91ce068537e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-def6e9dd-7hzmq" Jul 23 18:18:50.587964 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:50.587880 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-def6e9dd-7hzmq" Jul 23 18:18:50.727286 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:50.727258 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-def6e9dd-7hzmq"] Jul 23 18:18:50.729704 ip-10-0-137-82 kubenswrapper[2565]: W0723 18:18:50.729642 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6610e788_85c7_4a3c_aae5_d91ce068537e.slice/crio-c82991b9ac30dfdedf86bc50fcad6ab40a9133a8bf1a8dd031a17fd4bc521941 WatchSource:0}: Error finding container c82991b9ac30dfdedf86bc50fcad6ab40a9133a8bf1a8dd031a17fd4bc521941: Status 404 returned error can't find the container with id c82991b9ac30dfdedf86bc50fcad6ab40a9133a8bf1a8dd031a17fd4bc521941 Jul 23 18:18:51.268696 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:51.268662 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-def6e9dd-7hzmq" event={"ID":"6610e788-85c7-4a3c-aae5-d91ce068537e","Type":"ContainerStarted","Data":"69054b41636b31e85d11e6082a87ea2a7cd3bcbe8f0745f8a85b7e0a7a0b6f65"} Jul 23 18:18:51.268876 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:51.268718 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-def6e9dd-7hzmq" event={"ID":"6610e788-85c7-4a3c-aae5-d91ce068537e","Type":"ContainerStarted","Data":"c82991b9ac30dfdedf86bc50fcad6ab40a9133a8bf1a8dd031a17fd4bc521941"} Jul 23 18:18:51.286128 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:51.286079 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-def6e9dd-7hzmq" podStartSLOduration=1.2860622099999999 podStartE2EDuration="1.28606221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:18:51.284118092 +0000 UTC m=+2075.362421447" watchObservedRunningTime="2026-07-23 18:18:51.28606221 +0000 UTC m=+2075.364365562" Jul 23 18:18:52.274802 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:52.274763 2565 generic.go:356] "Generic (PLEG): container finished" podID="6610e788-85c7-4a3c-aae5-d91ce068537e" containerID="69054b41636b31e85d11e6082a87ea2a7cd3bcbe8f0745f8a85b7e0a7a0b6f65" exitCode=0 Jul 23 18:18:52.275004 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:52.274816 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-def6e9dd-7hzmq" event={"ID":"6610e788-85c7-4a3c-aae5-d91ce068537e","Type":"ContainerDied","Data":"69054b41636b31e85d11e6082a87ea2a7cd3bcbe8f0745f8a85b7e0a7a0b6f65"} Jul 23 18:18:53.419707 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:53.419677 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-def6e9dd-7hzmq" Jul 23 18:18:53.570293 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:53.570200 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6610e788-85c7-4a3c-aae5-d91ce068537e-maas-api-service-ca\") pod \"6610e788-85c7-4a3c-aae5-d91ce068537e\" (UID: \"6610e788-85c7-4a3c-aae5-d91ce068537e\") " Jul 23 18:18:53.570645 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:53.570590 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6610e788-85c7-4a3c-aae5-d91ce068537e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6610e788-85c7-4a3c-aae5-d91ce068537e" (UID: "6610e788-85c7-4a3c-aae5-d91ce068537e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:18:53.671300 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:53.671261 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6610e788-85c7-4a3c-aae5-d91ce068537e-maas-api-service-ca\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:18:54.286269 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:54.286235 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-def6e9dd-7hzmq" event={"ID":"6610e788-85c7-4a3c-aae5-d91ce068537e","Type":"ContainerDied","Data":"c82991b9ac30dfdedf86bc50fcad6ab40a9133a8bf1a8dd031a17fd4bc521941"} Jul 23 18:18:54.286439 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:54.286273 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-def6e9dd-7hzmq" Jul 23 18:18:54.286439 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:18:54.286284 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c82991b9ac30dfdedf86bc50fcad6ab40a9133a8bf1a8dd031a17fd4bc521941" Jul 23 18:19:05.880192 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:19:05.880154 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c"] Jul 23 18:19:05.880685 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:19:05.880421 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c" podUID="93dda095-70be-49a5-b461-5b2d5c282078" containerName="maas-api" containerID="cri-o://70e1fe422690e3fe4d4bd401b2c688171e3e62479b247dca4d2b805381378242" gracePeriod=30 Jul 23 18:19:06.137753 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:19:06.137679 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c" Jul 23 18:19:06.191308 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:19:06.191272 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93dda095-70be-49a5-b461-5b2d5c282078-openshift-service-ca\") pod \"93dda095-70be-49a5-b461-5b2d5c282078\" (UID: \"93dda095-70be-49a5-b461-5b2d5c282078\") " Jul 23 18:19:06.191476 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:19:06.191329 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zz7hf\" (UniqueName: \"kubernetes.io/projected/93dda095-70be-49a5-b461-5b2d5c282078-kube-api-access-zz7hf\") pod \"93dda095-70be-49a5-b461-5b2d5c282078\" (UID: \"93dda095-70be-49a5-b461-5b2d5c282078\") " Jul 23 18:19:06.191476 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:19:06.191420 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93dda095-70be-49a5-b461-5b2d5c282078-maas-api-tls\") pod \"93dda095-70be-49a5-b461-5b2d5c282078\" (UID: \"93dda095-70be-49a5-b461-5b2d5c282078\") " Jul 23 18:19:06.191698 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:19:06.191674 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93dda095-70be-49a5-b461-5b2d5c282078-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "93dda095-70be-49a5-b461-5b2d5c282078" (UID: "93dda095-70be-49a5-b461-5b2d5c282078"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:19:06.193529 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:19:06.193489 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93dda095-70be-49a5-b461-5b2d5c282078-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "93dda095-70be-49a5-b461-5b2d5c282078" (UID: "93dda095-70be-49a5-b461-5b2d5c282078"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:19:06.193529 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:19:06.193507 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93dda095-70be-49a5-b461-5b2d5c282078-kube-api-access-zz7hf" (OuterVolumeSpecName: "kube-api-access-zz7hf") pod "93dda095-70be-49a5-b461-5b2d5c282078" (UID: "93dda095-70be-49a5-b461-5b2d5c282078"). InnerVolumeSpecName "kube-api-access-zz7hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:19:06.292819 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:19:06.292789 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz7hf\" (UniqueName: \"kubernetes.io/projected/93dda095-70be-49a5-b461-5b2d5c282078-kube-api-access-zz7hf\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:19:06.292819 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:19:06.292816 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93dda095-70be-49a5-b461-5b2d5c282078-maas-api-tls\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:19:06.292819 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:19:06.292825 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93dda095-70be-49a5-b461-5b2d5c282078-openshift-service-ca\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:19:06.348510 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:19:06.348476 2565 generic.go:356] "Generic (PLEG): container finished" podID="93dda095-70be-49a5-b461-5b2d5c282078" containerID="70e1fe422690e3fe4d4bd401b2c688171e3e62479b247dca4d2b805381378242" exitCode=0 Jul 23 18:19:06.348510 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:19:06.348509 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c" Jul 23 18:19:06.348758 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:19:06.348524 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c" event={"ID":"93dda095-70be-49a5-b461-5b2d5c282078","Type":"ContainerDied","Data":"70e1fe422690e3fe4d4bd401b2c688171e3e62479b247dca4d2b805381378242"} Jul 23 18:19:06.348758 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:19:06.348566 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c" event={"ID":"93dda095-70be-49a5-b461-5b2d5c282078","Type":"ContainerDied","Data":"29bf884717420efe0e001f639e18153e880786343ae8cdb185b24efec57e60f0"} Jul 23 18:19:06.348758 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:19:06.348592 2565 scope.go:117] "RemoveContainer" containerID="70e1fe422690e3fe4d4bd401b2c688171e3e62479b247dca4d2b805381378242" Jul 23 18:19:06.357827 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:19:06.357804 2565 scope.go:117] "RemoveContainer" containerID="70e1fe422690e3fe4d4bd401b2c688171e3e62479b247dca4d2b805381378242" Jul 23 18:19:06.358099 ip-10-0-137-82 kubenswrapper[2565]: E0723 18:19:06.358072 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70e1fe422690e3fe4d4bd401b2c688171e3e62479b247dca4d2b805381378242\": container with ID starting with 70e1fe422690e3fe4d4bd401b2c688171e3e62479b247dca4d2b805381378242 not found: ID does not exist" containerID="70e1fe422690e3fe4d4bd401b2c688171e3e62479b247dca4d2b805381378242" Jul 23 18:19:06.358161 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:19:06.358113 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70e1fe422690e3fe4d4bd401b2c688171e3e62479b247dca4d2b805381378242"} err="failed to get container status \"70e1fe422690e3fe4d4bd401b2c688171e3e62479b247dca4d2b805381378242\": rpc error: code = NotFound desc = could not find container \"70e1fe422690e3fe4d4bd401b2c688171e3e62479b247dca4d2b805381378242\": container with ID starting with 70e1fe422690e3fe4d4bd401b2c688171e3e62479b247dca4d2b805381378242 not found: ID does not exist" Jul 23 18:19:06.373359 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:19:06.373336 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c"] Jul 23 18:19:06.376783 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:19:06.376752 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-def6e9dd-569fb8b785-56s7c"] Jul 23 18:19:06.406390 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:19:06.406307 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93dda095-70be-49a5-b461-5b2d5c282078" path="/var/lib/kubelet/pods/93dda095-70be-49a5-b461-5b2d5c282078/volumes" Jul 23 18:19:10.361050 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:19:10.361019 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-def6e9dd-7hzmq"] Jul 23 18:19:10.363958 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:19:10.363930 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-def6e9dd-7hzmq"] Jul 23 18:19:10.405576 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:19:10.405545 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6610e788-85c7-4a3c-aae5-d91ce068537e" path="/var/lib/kubelet/pods/6610e788-85c7-4a3c-aae5-d91ce068537e/volumes" Jul 23 18:19:16.485963 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:19:16.485933 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-m7k4j_1cca3236-5c90-4abb-9b9b-39e40b1903c3/console-operator/1.log" Jul 23 18:19:16.493382 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:19:16.493359 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-m7k4j_1cca3236-5c90-4abb-9b9b-39e40b1903c3/console-operator/1.log" Jul 23 18:21:00.073292 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:00.073249 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9"] Jul 23 18:21:00.073791 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:00.073780 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93dda095-70be-49a5-b461-5b2d5c282078" containerName="maas-api" Jul 23 18:21:00.073848 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:00.073795 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="93dda095-70be-49a5-b461-5b2d5c282078" containerName="maas-api" Jul 23 18:21:00.073848 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:00.073834 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6610e788-85c7-4a3c-aae5-d91ce068537e" containerName="revoke-keys" Jul 23 18:21:00.073848 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:00.073843 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6610e788-85c7-4a3c-aae5-d91ce068537e" containerName="revoke-keys" Jul 23 18:21:00.073995 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:00.073931 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6610e788-85c7-4a3c-aae5-d91ce068537e" containerName="revoke-keys" Jul 23 18:21:00.073995 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:00.073947 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="93dda095-70be-49a5-b461-5b2d5c282078" containerName="maas-api" Jul 23 18:21:00.077338 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:00.077318 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9" Jul 23 18:21:00.079796 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:00.079771 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-090448-serving-cert\"" Jul 23 18:21:00.087449 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:00.087422 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9"] Jul 23 18:21:00.097011 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:00.096977 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx8pf\" (UniqueName: \"kubernetes.io/projected/1cd6b07e-ff33-4e7f-b42d-de863992fd55-kube-api-access-bx8pf\") pod \"maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9\" (UID: \"1cd6b07e-ff33-4e7f-b42d-de863992fd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9" Jul 23 18:21:00.097158 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:00.097027 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cd6b07e-ff33-4e7f-b42d-de863992fd55-maas-api-tls\") pod \"maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9\" (UID: \"1cd6b07e-ff33-4e7f-b42d-de863992fd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9" Jul 23 18:21:00.097158 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:00.097069 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd6b07e-ff33-4e7f-b42d-de863992fd55-openshift-service-ca\") pod \"maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9\" (UID: \"1cd6b07e-ff33-4e7f-b42d-de863992fd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9" Jul 23 18:21:00.197844 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:00.197805 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx8pf\" (UniqueName: \"kubernetes.io/projected/1cd6b07e-ff33-4e7f-b42d-de863992fd55-kube-api-access-bx8pf\") pod \"maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9\" (UID: \"1cd6b07e-ff33-4e7f-b42d-de863992fd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9" Jul 23 18:21:00.197844 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:00.197851 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cd6b07e-ff33-4e7f-b42d-de863992fd55-maas-api-tls\") pod \"maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9\" (UID: \"1cd6b07e-ff33-4e7f-b42d-de863992fd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9" Jul 23 18:21:00.198124 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:00.197881 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd6b07e-ff33-4e7f-b42d-de863992fd55-openshift-service-ca\") pod \"maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9\" (UID: \"1cd6b07e-ff33-4e7f-b42d-de863992fd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9" Jul 23 18:21:00.198124 ip-10-0-137-82 kubenswrapper[2565]: E0723 18:21:00.198021 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-090448-serving-cert: secret "maas-api-e2e-inf-a-090448-serving-cert" not found Jul 23 18:21:00.198124 ip-10-0-137-82 kubenswrapper[2565]: E0723 18:21:00.198101 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd6b07e-ff33-4e7f-b42d-de863992fd55-maas-api-tls podName:1cd6b07e-ff33-4e7f-b42d-de863992fd55 nodeName:}" failed. No retries permitted until 2026-07-23 18:21:00.698085753 +0000 UTC m=+2204.776389083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1cd6b07e-ff33-4e7f-b42d-de863992fd55-maas-api-tls") pod "maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9" (UID: "1cd6b07e-ff33-4e7f-b42d-de863992fd55") : secret "maas-api-e2e-inf-a-090448-serving-cert" not found Jul 23 18:21:00.198529 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:00.198511 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd6b07e-ff33-4e7f-b42d-de863992fd55-openshift-service-ca\") pod \"maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9\" (UID: \"1cd6b07e-ff33-4e7f-b42d-de863992fd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9" Jul 23 18:21:00.209991 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:00.209957 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx8pf\" (UniqueName: \"kubernetes.io/projected/1cd6b07e-ff33-4e7f-b42d-de863992fd55-kube-api-access-bx8pf\") pod \"maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9\" (UID: \"1cd6b07e-ff33-4e7f-b42d-de863992fd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9" Jul 23 18:21:00.702492 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:00.702456 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cd6b07e-ff33-4e7f-b42d-de863992fd55-maas-api-tls\") pod \"maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9\" (UID: \"1cd6b07e-ff33-4e7f-b42d-de863992fd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9" Jul 23 18:21:00.704898 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:00.704875 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cd6b07e-ff33-4e7f-b42d-de863992fd55-maas-api-tls\") pod \"maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9\" (UID: \"1cd6b07e-ff33-4e7f-b42d-de863992fd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9" Jul 23 18:21:00.989761 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:00.989660 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9" Jul 23 18:21:01.129204 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:01.129175 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9"] Jul 23 18:21:01.131006 ip-10-0-137-82 kubenswrapper[2565]: W0723 18:21:01.130980 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd6b07e_ff33_4e7f_b42d_de863992fd55.slice/crio-b87d5462ebc511e26b983fd71d1a584ad05fde343d57edf04ec7bb5c60f5e0f8 WatchSource:0}: Error finding container b87d5462ebc511e26b983fd71d1a584ad05fde343d57edf04ec7bb5c60f5e0f8: Status 404 returned error can't find the container with id b87d5462ebc511e26b983fd71d1a584ad05fde343d57edf04ec7bb5c60f5e0f8 Jul 23 18:21:01.927916 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:01.927821 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9" event={"ID":"1cd6b07e-ff33-4e7f-b42d-de863992fd55","Type":"ContainerStarted","Data":"549970cbc4a1c444dd2a34626199d233c815bae3014cb3fd22fcf9ec51698353"} Jul 23 18:21:01.928089 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:01.927926 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9" event={"ID":"1cd6b07e-ff33-4e7f-b42d-de863992fd55","Type":"ContainerStarted","Data":"b87d5462ebc511e26b983fd71d1a584ad05fde343d57edf04ec7bb5c60f5e0f8"} Jul 23 18:21:01.928089 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:01.927956 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9" Jul 23 18:21:01.948460 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:01.948404 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9" podStartSLOduration=1.409847794 podStartE2EDuration="1.9483887s" totalImagesPullingTime="538.540906ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:21:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:21:01.945778751 +0000 UTC m=+2206.024082104" watchObservedRunningTime="2026-07-23 18:21:01.9483887 +0000 UTC m=+2206.026692053" Jul 23 18:21:07.940536 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:07.940496 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9" Jul 23 18:21:08.294661 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:08.294620 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7"] Jul 23 18:21:08.298546 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:08.298521 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7" Jul 23 18:21:08.301206 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:08.301177 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-7893d7-serving-cert\"" Jul 23 18:21:08.309012 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:08.308983 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7"] Jul 23 18:21:08.366407 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:08.366365 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh6sd\" (UniqueName: \"kubernetes.io/projected/6d6bf10b-0c46-493f-ab50-fd7ca04444a9-kube-api-access-mh6sd\") pod \"maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7\" (UID: \"6d6bf10b-0c46-493f-ab50-fd7ca04444a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7" Jul 23 18:21:08.366648 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:08.366487 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d6bf10b-0c46-493f-ab50-fd7ca04444a9-maas-api-tls\") pod \"maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7\" (UID: \"6d6bf10b-0c46-493f-ab50-fd7ca04444a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7" Jul 23 18:21:08.366648 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:08.366520 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d6bf10b-0c46-493f-ab50-fd7ca04444a9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7\" (UID: \"6d6bf10b-0c46-493f-ab50-fd7ca04444a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7" Jul 23 18:21:08.467656 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:08.467619 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh6sd\" (UniqueName: \"kubernetes.io/projected/6d6bf10b-0c46-493f-ab50-fd7ca04444a9-kube-api-access-mh6sd\") pod \"maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7\" (UID: \"6d6bf10b-0c46-493f-ab50-fd7ca04444a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7" Jul 23 18:21:08.467836 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:08.467729 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d6bf10b-0c46-493f-ab50-fd7ca04444a9-maas-api-tls\") pod \"maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7\" (UID: \"6d6bf10b-0c46-493f-ab50-fd7ca04444a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7" Jul 23 18:21:08.467836 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:08.467752 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d6bf10b-0c46-493f-ab50-fd7ca04444a9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7\" (UID: \"6d6bf10b-0c46-493f-ab50-fd7ca04444a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7" Jul 23 18:21:08.468494 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:08.468473 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d6bf10b-0c46-493f-ab50-fd7ca04444a9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7\" (UID: \"6d6bf10b-0c46-493f-ab50-fd7ca04444a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7" Jul 23 18:21:08.470282 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:08.470261 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d6bf10b-0c46-493f-ab50-fd7ca04444a9-maas-api-tls\") pod \"maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7\" (UID: \"6d6bf10b-0c46-493f-ab50-fd7ca04444a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7" Jul 23 18:21:08.476020 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:08.475995 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh6sd\" (UniqueName: \"kubernetes.io/projected/6d6bf10b-0c46-493f-ab50-fd7ca04444a9-kube-api-access-mh6sd\") pod \"maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7\" (UID: \"6d6bf10b-0c46-493f-ab50-fd7ca04444a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7" Jul 23 18:21:08.612030 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:08.611940 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7" Jul 23 18:21:08.755291 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:08.755254 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7"] Jul 23 18:21:08.758949 ip-10-0-137-82 kubenswrapper[2565]: W0723 18:21:08.758918 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d6bf10b_0c46_493f_ab50_fd7ca04444a9.slice/crio-98757ff22a62e0fa2b16dd36613ec8bd91a6f3e6911fad2e1679e3d81e819096 WatchSource:0}: Error finding container 98757ff22a62e0fa2b16dd36613ec8bd91a6f3e6911fad2e1679e3d81e819096: Status 404 returned error can't find the container with id 98757ff22a62e0fa2b16dd36613ec8bd91a6f3e6911fad2e1679e3d81e819096 Jul 23 18:21:08.964761 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:08.964671 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7" event={"ID":"6d6bf10b-0c46-493f-ab50-fd7ca04444a9","Type":"ContainerStarted","Data":"98757ff22a62e0fa2b16dd36613ec8bd91a6f3e6911fad2e1679e3d81e819096"} Jul 23 18:21:09.971789 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:09.971755 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7" event={"ID":"6d6bf10b-0c46-493f-ab50-fd7ca04444a9","Type":"ContainerStarted","Data":"d8a98d13d5842c6ff2d154ccdb9f1a95f30bce9597afd1f4c1281e5917c877f6"} Jul 23 18:21:09.972171 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:09.971824 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7" Jul 23 18:21:09.990669 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:09.990625 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7" podStartSLOduration=1.672967641 podStartE2EDuration="1.990580821s" totalImagesPullingTime="317.61318ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:21:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:21:09.989792265 +0000 UTC m=+2214.068095620" watchObservedRunningTime="2026-07-23 18:21:09.990580821 +0000 UTC m=+2214.068884174" Jul 23 18:21:15.983628 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:15.983583 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7" Jul 23 18:21:16.594805 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:16.594775 2565 scope.go:117] "RemoveContainer" containerID="7a232e6ab24ee93a0a3a09288f2266495b594b429b59f7cc1aab34b71ee40e22" Jul 23 18:21:16.602945 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:16.602923 2565 scope.go:117] "RemoveContainer" containerID="0ff2d94749749ed6e70cf311bf163efb0e94ccee2e551a7b634cc03f9a6ac80a" Jul 23 18:21:16.610394 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:21:16.610376 2565 scope.go:117] "RemoveContainer" containerID="b60f6dca865d3b5cf4ba6bd3cea14db15c436a6a5801b5022bc8b0a083c7690d" Jul 23 18:22:38.790474 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:38.790429 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-090448-ch82t"] Jul 23 18:22:38.794074 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:38.794049 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-090448-ch82t" Jul 23 18:22:38.798083 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:38.798061 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5lmm\"" Jul 23 18:22:38.809738 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:38.809712 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-090448-ch82t"] Jul 23 18:22:38.891439 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:38.891402 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3e9f635-f895-4f3c-a88e-e4b0c6fd4ebf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-090448-ch82t\" (UID: \"e3e9f635-f895-4f3c-a88e-e4b0c6fd4ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-090448-ch82t" Jul 23 18:22:38.992368 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:38.992326 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3e9f635-f895-4f3c-a88e-e4b0c6fd4ebf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-090448-ch82t\" (UID: \"e3e9f635-f895-4f3c-a88e-e4b0c6fd4ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-090448-ch82t" Jul 23 18:22:38.992994 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:38.992972 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3e9f635-f895-4f3c-a88e-e4b0c6fd4ebf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-090448-ch82t\" (UID: \"e3e9f635-f895-4f3c-a88e-e4b0c6fd4ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-090448-ch82t" Jul 23 18:22:39.103872 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:39.103785 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-090448-ch82t" Jul 23 18:22:39.247755 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:39.247730 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-090448-ch82t"] Jul 23 18:22:39.250658 ip-10-0-137-82 kubenswrapper[2565]: W0723 18:22:39.250626 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3e9f635_f895_4f3c_a88e_e4b0c6fd4ebf.slice/crio-088ad96c49f806f8c0c5fceb677a93b9a4ea6af4cac5d558cd5684c8709e38a4 WatchSource:0}: Error finding container 088ad96c49f806f8c0c5fceb677a93b9a4ea6af4cac5d558cd5684c8709e38a4: Status 404 returned error can't find the container with id 088ad96c49f806f8c0c5fceb677a93b9a4ea6af4cac5d558cd5684c8709e38a4 Jul 23 18:22:39.423104 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:39.423026 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-090448-ch82t" event={"ID":"e3e9f635-f895-4f3c-a88e-e4b0c6fd4ebf","Type":"ContainerStarted","Data":"088ad96c49f806f8c0c5fceb677a93b9a4ea6af4cac5d558cd5684c8709e38a4"} Jul 23 18:22:40.428922 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:40.428886 2565 generic.go:356] "Generic (PLEG): container finished" podID="e3e9f635-f895-4f3c-a88e-e4b0c6fd4ebf" containerID="4833380c49c39ae6c21ccbfdb5b973f4a4f5ff8539784fcb2891678f20e36b81" exitCode=0 Jul 23 18:22:40.429311 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:40.428971 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-090448-ch82t" event={"ID":"e3e9f635-f895-4f3c-a88e-e4b0c6fd4ebf","Type":"ContainerDied","Data":"4833380c49c39ae6c21ccbfdb5b973f4a4f5ff8539784fcb2891678f20e36b81"} Jul 23 18:22:41.570943 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:41.570918 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-090448-ch82t" Jul 23 18:22:41.617058 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:41.617031 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3e9f635-f895-4f3c-a88e-e4b0c6fd4ebf-maas-api-service-ca\") pod \"e3e9f635-f895-4f3c-a88e-e4b0c6fd4ebf\" (UID: \"e3e9f635-f895-4f3c-a88e-e4b0c6fd4ebf\") " Jul 23 18:22:41.617385 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:41.617362 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3e9f635-f895-4f3c-a88e-e4b0c6fd4ebf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e3e9f635-f895-4f3c-a88e-e4b0c6fd4ebf" (UID: "e3e9f635-f895-4f3c-a88e-e4b0c6fd4ebf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:22:41.718404 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:41.718321 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3e9f635-f895-4f3c-a88e-e4b0c6fd4ebf-maas-api-service-ca\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:22:42.440689 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:42.440659 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-090448-ch82t" Jul 23 18:22:42.440689 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:42.440680 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-090448-ch82t" event={"ID":"e3e9f635-f895-4f3c-a88e-e4b0c6fd4ebf","Type":"ContainerDied","Data":"088ad96c49f806f8c0c5fceb677a93b9a4ea6af4cac5d558cd5684c8709e38a4"} Jul 23 18:22:42.440888 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:42.440727 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="088ad96c49f806f8c0c5fceb677a93b9a4ea6af4cac5d558cd5684c8709e38a4" Jul 23 18:22:49.436081 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:49.436042 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9"] Jul 23 18:22:49.436988 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:49.436935 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9" podUID="1cd6b07e-ff33-4e7f-b42d-de863992fd55" containerName="maas-api" containerID="cri-o://549970cbc4a1c444dd2a34626199d233c815bae3014cb3fd22fcf9ec51698353" gracePeriod=30 Jul 23 18:22:49.695372 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:49.695307 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9" Jul 23 18:22:49.786943 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:49.786907 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cd6b07e-ff33-4e7f-b42d-de863992fd55-maas-api-tls\") pod \"1cd6b07e-ff33-4e7f-b42d-de863992fd55\" (UID: \"1cd6b07e-ff33-4e7f-b42d-de863992fd55\") " Jul 23 18:22:49.787114 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:49.786962 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd6b07e-ff33-4e7f-b42d-de863992fd55-openshift-service-ca\") pod \"1cd6b07e-ff33-4e7f-b42d-de863992fd55\" (UID: \"1cd6b07e-ff33-4e7f-b42d-de863992fd55\") " Jul 23 18:22:49.787114 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:49.786990 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bx8pf\" (UniqueName: \"kubernetes.io/projected/1cd6b07e-ff33-4e7f-b42d-de863992fd55-kube-api-access-bx8pf\") pod \"1cd6b07e-ff33-4e7f-b42d-de863992fd55\" (UID: \"1cd6b07e-ff33-4e7f-b42d-de863992fd55\") " Jul 23 18:22:49.787374 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:49.787334 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd6b07e-ff33-4e7f-b42d-de863992fd55-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1cd6b07e-ff33-4e7f-b42d-de863992fd55" (UID: "1cd6b07e-ff33-4e7f-b42d-de863992fd55"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:22:49.789100 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:49.789069 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd6b07e-ff33-4e7f-b42d-de863992fd55-kube-api-access-bx8pf" (OuterVolumeSpecName: "kube-api-access-bx8pf") pod "1cd6b07e-ff33-4e7f-b42d-de863992fd55" (UID: "1cd6b07e-ff33-4e7f-b42d-de863992fd55"). InnerVolumeSpecName "kube-api-access-bx8pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:22:49.789193 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:49.789093 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cd6b07e-ff33-4e7f-b42d-de863992fd55-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1cd6b07e-ff33-4e7f-b42d-de863992fd55" (UID: "1cd6b07e-ff33-4e7f-b42d-de863992fd55"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:22:49.887711 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:49.887675 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cd6b07e-ff33-4e7f-b42d-de863992fd55-maas-api-tls\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:22:49.887711 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:49.887706 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd6b07e-ff33-4e7f-b42d-de863992fd55-openshift-service-ca\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:22:49.887711 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:49.887718 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx8pf\" (UniqueName: \"kubernetes.io/projected/1cd6b07e-ff33-4e7f-b42d-de863992fd55-kube-api-access-bx8pf\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:22:50.482507 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:50.482469 2565 generic.go:356] "Generic (PLEG): container finished" podID="1cd6b07e-ff33-4e7f-b42d-de863992fd55" containerID="549970cbc4a1c444dd2a34626199d233c815bae3014cb3fd22fcf9ec51698353" exitCode=0 Jul 23 18:22:50.482963 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:50.482515 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9" Jul 23 18:22:50.482963 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:50.482552 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9" event={"ID":"1cd6b07e-ff33-4e7f-b42d-de863992fd55","Type":"ContainerDied","Data":"549970cbc4a1c444dd2a34626199d233c815bae3014cb3fd22fcf9ec51698353"} Jul 23 18:22:50.482963 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:50.482627 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9" event={"ID":"1cd6b07e-ff33-4e7f-b42d-de863992fd55","Type":"ContainerDied","Data":"b87d5462ebc511e26b983fd71d1a584ad05fde343d57edf04ec7bb5c60f5e0f8"} Jul 23 18:22:50.482963 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:50.482650 2565 scope.go:117] "RemoveContainer" containerID="549970cbc4a1c444dd2a34626199d233c815bae3014cb3fd22fcf9ec51698353" Jul 23 18:22:50.491712 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:50.491589 2565 scope.go:117] "RemoveContainer" containerID="549970cbc4a1c444dd2a34626199d233c815bae3014cb3fd22fcf9ec51698353" Jul 23 18:22:50.491928 ip-10-0-137-82 kubenswrapper[2565]: E0723 18:22:50.491908 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"549970cbc4a1c444dd2a34626199d233c815bae3014cb3fd22fcf9ec51698353\": container with ID starting with 549970cbc4a1c444dd2a34626199d233c815bae3014cb3fd22fcf9ec51698353 not found: ID does not exist" containerID="549970cbc4a1c444dd2a34626199d233c815bae3014cb3fd22fcf9ec51698353" Jul 23 18:22:50.492010 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:50.491947 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"549970cbc4a1c444dd2a34626199d233c815bae3014cb3fd22fcf9ec51698353"} err="failed to get container status \"549970cbc4a1c444dd2a34626199d233c815bae3014cb3fd22fcf9ec51698353\": rpc error: code = NotFound desc = could not find container \"549970cbc4a1c444dd2a34626199d233c815bae3014cb3fd22fcf9ec51698353\": container with ID starting with 549970cbc4a1c444dd2a34626199d233c815bae3014cb3fd22fcf9ec51698353 not found: ID does not exist" Jul 23 18:22:50.506389 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:50.506360 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9"] Jul 23 18:22:50.512611 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:50.512570 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-090448-77dd5fdbf5-4fvw9"] Jul 23 18:22:52.405895 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:52.405847 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cd6b07e-ff33-4e7f-b42d-de863992fd55" path="/var/lib/kubelet/pods/1cd6b07e-ff33-4e7f-b42d-de863992fd55/volumes" Jul 23 18:22:53.938781 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:53.938744 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-090448-ch82t"] Jul 23 18:22:53.944112 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:53.944075 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-090448-ch82t"] Jul 23 18:22:54.405568 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:22:54.405536 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3e9f635-f895-4f3c-a88e-e4b0c6fd4ebf" path="/var/lib/kubelet/pods/e3e9f635-f895-4f3c-a88e-e4b0c6fd4ebf/volumes" Jul 23 18:23:01.058893 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:01.058855 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7893d7-d75zb"] Jul 23 18:23:01.059287 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:01.059273 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3e9f635-f895-4f3c-a88e-e4b0c6fd4ebf" containerName="revoke-keys" Jul 23 18:23:01.059331 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:01.059289 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3e9f635-f895-4f3c-a88e-e4b0c6fd4ebf" containerName="revoke-keys" Jul 23 18:23:01.059331 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:01.059312 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cd6b07e-ff33-4e7f-b42d-de863992fd55" containerName="maas-api" Jul 23 18:23:01.059331 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:01.059318 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cd6b07e-ff33-4e7f-b42d-de863992fd55" containerName="maas-api" Jul 23 18:23:01.059432 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:01.059381 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3e9f635-f895-4f3c-a88e-e4b0c6fd4ebf" containerName="revoke-keys" Jul 23 18:23:01.059432 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:01.059391 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cd6b07e-ff33-4e7f-b42d-de863992fd55" containerName="maas-api" Jul 23 18:23:01.063454 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:01.063438 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7893d7-d75zb" Jul 23 18:23:01.067350 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:01.067331 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5lmm\"" Jul 23 18:23:01.075030 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:01.075005 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7893d7-d75zb"] Jul 23 18:23:01.192451 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:01.192410 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/689f8e1a-2eb7-4cbd-a728-05047e04340a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7893d7-d75zb\" (UID: \"689f8e1a-2eb7-4cbd-a728-05047e04340a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7893d7-d75zb" Jul 23 18:23:01.293663 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:01.293625 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/689f8e1a-2eb7-4cbd-a728-05047e04340a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7893d7-d75zb\" (UID: \"689f8e1a-2eb7-4cbd-a728-05047e04340a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7893d7-d75zb" Jul 23 18:23:01.294314 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:01.294287 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/689f8e1a-2eb7-4cbd-a728-05047e04340a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7893d7-d75zb\" (UID: \"689f8e1a-2eb7-4cbd-a728-05047e04340a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7893d7-d75zb" Jul 23 18:23:01.373365 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:01.373279 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7893d7-d75zb" Jul 23 18:23:01.544931 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:01.544906 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7893d7-d75zb"] Jul 23 18:23:01.546938 ip-10-0-137-82 kubenswrapper[2565]: W0723 18:23:01.546911 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod689f8e1a_2eb7_4cbd_a728_05047e04340a.slice/crio-58367da68be457fdb937b19590f21aa2b17b6b969f427ddab26c852c7ea3e6e8 WatchSource:0}: Error finding container 58367da68be457fdb937b19590f21aa2b17b6b969f427ddab26c852c7ea3e6e8: Status 404 returned error can't find the container with id 58367da68be457fdb937b19590f21aa2b17b6b969f427ddab26c852c7ea3e6e8 Jul 23 18:23:02.543049 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:02.543009 2565 generic.go:356] "Generic (PLEG): container finished" podID="689f8e1a-2eb7-4cbd-a728-05047e04340a" containerID="45178ba2c28966632b4abc900c65b2fed0960b881604f12546e41ad2c30bafd5" exitCode=0 Jul 23 18:23:02.543049 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:02.543048 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7893d7-d75zb" event={"ID":"689f8e1a-2eb7-4cbd-a728-05047e04340a","Type":"ContainerDied","Data":"45178ba2c28966632b4abc900c65b2fed0960b881604f12546e41ad2c30bafd5"} Jul 23 18:23:02.543536 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:02.543084 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7893d7-d75zb" event={"ID":"689f8e1a-2eb7-4cbd-a728-05047e04340a","Type":"ContainerStarted","Data":"58367da68be457fdb937b19590f21aa2b17b6b969f427ddab26c852c7ea3e6e8"} Jul 23 18:23:03.685830 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:03.685800 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7893d7-d75zb" Jul 23 18:23:03.818986 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:03.818882 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/689f8e1a-2eb7-4cbd-a728-05047e04340a-maas-api-service-ca\") pod \"689f8e1a-2eb7-4cbd-a728-05047e04340a\" (UID: \"689f8e1a-2eb7-4cbd-a728-05047e04340a\") " Jul 23 18:23:03.819267 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:03.819243 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/689f8e1a-2eb7-4cbd-a728-05047e04340a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "689f8e1a-2eb7-4cbd-a728-05047e04340a" (UID: "689f8e1a-2eb7-4cbd-a728-05047e04340a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:23:03.919792 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:03.919755 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/689f8e1a-2eb7-4cbd-a728-05047e04340a-maas-api-service-ca\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:23:04.555791 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:04.555756 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7893d7-d75zb" Jul 23 18:23:04.555791 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:04.555787 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7893d7-d75zb" event={"ID":"689f8e1a-2eb7-4cbd-a728-05047e04340a","Type":"ContainerDied","Data":"58367da68be457fdb937b19590f21aa2b17b6b969f427ddab26c852c7ea3e6e8"} Jul 23 18:23:04.556046 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:04.555835 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58367da68be457fdb937b19590f21aa2b17b6b969f427ddab26c852c7ea3e6e8" Jul 23 18:23:11.718004 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:11.717918 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7"] Jul 23 18:23:11.718436 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:11.718286 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7" podUID="6d6bf10b-0c46-493f-ab50-fd7ca04444a9" containerName="maas-api" containerID="cri-o://d8a98d13d5842c6ff2d154ccdb9f1a95f30bce9597afd1f4c1281e5917c877f6" gracePeriod=30 Jul 23 18:23:11.972049 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:11.971982 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7" Jul 23 18:23:12.097539 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:12.097500 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mh6sd\" (UniqueName: \"kubernetes.io/projected/6d6bf10b-0c46-493f-ab50-fd7ca04444a9-kube-api-access-mh6sd\") pod \"6d6bf10b-0c46-493f-ab50-fd7ca04444a9\" (UID: \"6d6bf10b-0c46-493f-ab50-fd7ca04444a9\") " Jul 23 18:23:12.097743 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:12.097550 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d6bf10b-0c46-493f-ab50-fd7ca04444a9-openshift-service-ca\") pod \"6d6bf10b-0c46-493f-ab50-fd7ca04444a9\" (UID: \"6d6bf10b-0c46-493f-ab50-fd7ca04444a9\") " Jul 23 18:23:12.097743 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:12.097588 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d6bf10b-0c46-493f-ab50-fd7ca04444a9-maas-api-tls\") pod \"6d6bf10b-0c46-493f-ab50-fd7ca04444a9\" (UID: \"6d6bf10b-0c46-493f-ab50-fd7ca04444a9\") " Jul 23 18:23:12.097951 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:12.097918 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d6bf10b-0c46-493f-ab50-fd7ca04444a9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6d6bf10b-0c46-493f-ab50-fd7ca04444a9" (UID: "6d6bf10b-0c46-493f-ab50-fd7ca04444a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:23:12.099721 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:12.099690 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d6bf10b-0c46-493f-ab50-fd7ca04444a9-kube-api-access-mh6sd" (OuterVolumeSpecName: "kube-api-access-mh6sd") pod "6d6bf10b-0c46-493f-ab50-fd7ca04444a9" (UID: "6d6bf10b-0c46-493f-ab50-fd7ca04444a9"). InnerVolumeSpecName "kube-api-access-mh6sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:23:12.099806 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:12.099702 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d6bf10b-0c46-493f-ab50-fd7ca04444a9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6d6bf10b-0c46-493f-ab50-fd7ca04444a9" (UID: "6d6bf10b-0c46-493f-ab50-fd7ca04444a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:23:12.198870 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:12.198827 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d6bf10b-0c46-493f-ab50-fd7ca04444a9-maas-api-tls\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:23:12.198870 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:12.198862 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh6sd\" (UniqueName: \"kubernetes.io/projected/6d6bf10b-0c46-493f-ab50-fd7ca04444a9-kube-api-access-mh6sd\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:23:12.198870 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:12.198872 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d6bf10b-0c46-493f-ab50-fd7ca04444a9-openshift-service-ca\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:23:12.597189 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:12.597153 2565 generic.go:356] "Generic (PLEG): container finished" podID="6d6bf10b-0c46-493f-ab50-fd7ca04444a9" containerID="d8a98d13d5842c6ff2d154ccdb9f1a95f30bce9597afd1f4c1281e5917c877f6" exitCode=0 Jul 23 18:23:12.597189 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:12.597193 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7" Jul 23 18:23:12.597418 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:12.597238 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7" event={"ID":"6d6bf10b-0c46-493f-ab50-fd7ca04444a9","Type":"ContainerDied","Data":"d8a98d13d5842c6ff2d154ccdb9f1a95f30bce9597afd1f4c1281e5917c877f6"} Jul 23 18:23:12.597418 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:12.597288 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7" event={"ID":"6d6bf10b-0c46-493f-ab50-fd7ca04444a9","Type":"ContainerDied","Data":"98757ff22a62e0fa2b16dd36613ec8bd91a6f3e6911fad2e1679e3d81e819096"} Jul 23 18:23:12.597418 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:12.597309 2565 scope.go:117] "RemoveContainer" containerID="d8a98d13d5842c6ff2d154ccdb9f1a95f30bce9597afd1f4c1281e5917c877f6" Jul 23 18:23:12.606234 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:12.606217 2565 scope.go:117] "RemoveContainer" containerID="d8a98d13d5842c6ff2d154ccdb9f1a95f30bce9597afd1f4c1281e5917c877f6" Jul 23 18:23:12.606489 ip-10-0-137-82 kubenswrapper[2565]: E0723 18:23:12.606471 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8a98d13d5842c6ff2d154ccdb9f1a95f30bce9597afd1f4c1281e5917c877f6\": container with ID starting with d8a98d13d5842c6ff2d154ccdb9f1a95f30bce9597afd1f4c1281e5917c877f6 not found: ID does not exist" containerID="d8a98d13d5842c6ff2d154ccdb9f1a95f30bce9597afd1f4c1281e5917c877f6" Jul 23 18:23:12.606536 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:12.606499 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a98d13d5842c6ff2d154ccdb9f1a95f30bce9597afd1f4c1281e5917c877f6"} err="failed to get container status \"d8a98d13d5842c6ff2d154ccdb9f1a95f30bce9597afd1f4c1281e5917c877f6\": rpc error: code = NotFound desc = could not find container \"d8a98d13d5842c6ff2d154ccdb9f1a95f30bce9597afd1f4c1281e5917c877f6\": container with ID starting with d8a98d13d5842c6ff2d154ccdb9f1a95f30bce9597afd1f4c1281e5917c877f6 not found: ID does not exist" Jul 23 18:23:12.618905 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:12.618873 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7"] Jul 23 18:23:12.621942 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:12.621920 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7893d7-7b954dfd49-l9dk7"] Jul 23 18:23:14.407018 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:14.406978 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d6bf10b-0c46-493f-ab50-fd7ca04444a9" path="/var/lib/kubelet/pods/6d6bf10b-0c46-493f-ab50-fd7ca04444a9/volumes" Jul 23 18:23:16.186358 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:16.186322 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7893d7-d75zb"] Jul 23 18:23:16.190706 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:16.190682 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7893d7-d75zb"] Jul 23 18:23:16.406872 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:16.406839 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="689f8e1a-2eb7-4cbd-a728-05047e04340a" path="/var/lib/kubelet/pods/689f8e1a-2eb7-4cbd-a728-05047e04340a/volumes" Jul 23 18:23:27.218556 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:27.218518 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b34428-64548bc84-gqv2z"] Jul 23 18:23:27.219202 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:27.219103 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d6bf10b-0c46-493f-ab50-fd7ca04444a9" containerName="maas-api" Jul 23 18:23:27.219202 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:27.219119 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d6bf10b-0c46-493f-ab50-fd7ca04444a9" containerName="maas-api" Jul 23 18:23:27.219202 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:27.219137 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="689f8e1a-2eb7-4cbd-a728-05047e04340a" containerName="revoke-keys" Jul 23 18:23:27.219202 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:27.219145 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="689f8e1a-2eb7-4cbd-a728-05047e04340a" containerName="revoke-keys" Jul 23 18:23:27.219457 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:27.219244 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d6bf10b-0c46-493f-ab50-fd7ca04444a9" containerName="maas-api" Jul 23 18:23:27.219457 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:27.219262 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="689f8e1a-2eb7-4cbd-a728-05047e04340a" containerName="revoke-keys" Jul 23 18:23:27.222822 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:27.222799 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34428-64548bc84-gqv2z" Jul 23 18:23:27.226902 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:27.226864 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-b34428-serving-cert\"" Jul 23 18:23:27.231870 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:27.231841 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f72469-a821-4f59-91b7-25917a592d8e-openshift-service-ca\") pod \"maas-api-e2e-api-a-b34428-64548bc84-gqv2z\" (UID: \"78f72469-a821-4f59-91b7-25917a592d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34428-64548bc84-gqv2z" Jul 23 18:23:27.231980 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:27.231879 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78f72469-a821-4f59-91b7-25917a592d8e-maas-api-tls\") pod \"maas-api-e2e-api-a-b34428-64548bc84-gqv2z\" (UID: \"78f72469-a821-4f59-91b7-25917a592d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34428-64548bc84-gqv2z" Jul 23 18:23:27.231980 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:27.231908 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vw68\" (UniqueName: \"kubernetes.io/projected/78f72469-a821-4f59-91b7-25917a592d8e-kube-api-access-8vw68\") pod \"maas-api-e2e-api-a-b34428-64548bc84-gqv2z\" (UID: \"78f72469-a821-4f59-91b7-25917a592d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34428-64548bc84-gqv2z" Jul 23 18:23:27.328828 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:27.328769 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b34428-64548bc84-gqv2z"] Jul 23 18:23:27.332583 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:27.332550 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f72469-a821-4f59-91b7-25917a592d8e-openshift-service-ca\") pod \"maas-api-e2e-api-a-b34428-64548bc84-gqv2z\" (UID: \"78f72469-a821-4f59-91b7-25917a592d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34428-64548bc84-gqv2z" Jul 23 18:23:27.332751 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:27.332655 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78f72469-a821-4f59-91b7-25917a592d8e-maas-api-tls\") pod \"maas-api-e2e-api-a-b34428-64548bc84-gqv2z\" (UID: \"78f72469-a821-4f59-91b7-25917a592d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34428-64548bc84-gqv2z" Jul 23 18:23:27.332751 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:27.332689 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vw68\" (UniqueName: \"kubernetes.io/projected/78f72469-a821-4f59-91b7-25917a592d8e-kube-api-access-8vw68\") pod \"maas-api-e2e-api-a-b34428-64548bc84-gqv2z\" (UID: \"78f72469-a821-4f59-91b7-25917a592d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34428-64548bc84-gqv2z" Jul 23 18:23:27.333121 ip-10-0-137-82 kubenswrapper[2565]: E0723 18:23:27.333101 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-b34428-serving-cert: secret "maas-api-e2e-api-a-b34428-serving-cert" not found Jul 23 18:23:27.333215 ip-10-0-137-82 kubenswrapper[2565]: E0723 18:23:27.333203 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78f72469-a821-4f59-91b7-25917a592d8e-maas-api-tls podName:78f72469-a821-4f59-91b7-25917a592d8e nodeName:}" failed. No retries permitted until 2026-07-23 18:23:27.833180845 +0000 UTC m=+2351.911484201 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/78f72469-a821-4f59-91b7-25917a592d8e-maas-api-tls") pod "maas-api-e2e-api-a-b34428-64548bc84-gqv2z" (UID: "78f72469-a821-4f59-91b7-25917a592d8e") : secret "maas-api-e2e-api-a-b34428-serving-cert" not found Jul 23 18:23:27.333322 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:27.333302 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f72469-a821-4f59-91b7-25917a592d8e-openshift-service-ca\") pod \"maas-api-e2e-api-a-b34428-64548bc84-gqv2z\" (UID: \"78f72469-a821-4f59-91b7-25917a592d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34428-64548bc84-gqv2z" Jul 23 18:23:27.355384 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:27.355361 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vw68\" (UniqueName: \"kubernetes.io/projected/78f72469-a821-4f59-91b7-25917a592d8e-kube-api-access-8vw68\") pod \"maas-api-e2e-api-a-b34428-64548bc84-gqv2z\" (UID: \"78f72469-a821-4f59-91b7-25917a592d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34428-64548bc84-gqv2z" Jul 23 18:23:27.835985 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:27.835944 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78f72469-a821-4f59-91b7-25917a592d8e-maas-api-tls\") pod \"maas-api-e2e-api-a-b34428-64548bc84-gqv2z\" (UID: \"78f72469-a821-4f59-91b7-25917a592d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34428-64548bc84-gqv2z" Jul 23 18:23:27.838314 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:27.838293 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78f72469-a821-4f59-91b7-25917a592d8e-maas-api-tls\") pod \"maas-api-e2e-api-a-b34428-64548bc84-gqv2z\" (UID: \"78f72469-a821-4f59-91b7-25917a592d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34428-64548bc84-gqv2z" Jul 23 18:23:28.133831 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:28.133744 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34428-64548bc84-gqv2z" Jul 23 18:23:28.279231 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:28.279202 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b34428-64548bc84-gqv2z"] Jul 23 18:23:28.282204 ip-10-0-137-82 kubenswrapper[2565]: W0723 18:23:28.282175 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f72469_a821_4f59_91b7_25917a592d8e.slice/crio-4315433aa78674e45561beee2dc276378fe8afeadffd096e5eeb7ef8d9e01fa0 WatchSource:0}: Error finding container 4315433aa78674e45561beee2dc276378fe8afeadffd096e5eeb7ef8d9e01fa0: Status 404 returned error can't find the container with id 4315433aa78674e45561beee2dc276378fe8afeadffd096e5eeb7ef8d9e01fa0 Jul 23 18:23:28.283472 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:28.283452 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 18:23:28.678335 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:28.678295 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34428-64548bc84-gqv2z" event={"ID":"78f72469-a821-4f59-91b7-25917a592d8e","Type":"ContainerStarted","Data":"4315433aa78674e45561beee2dc276378fe8afeadffd096e5eeb7ef8d9e01fa0"} Jul 23 18:23:29.685725 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:29.685693 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34428-64548bc84-gqv2z" event={"ID":"78f72469-a821-4f59-91b7-25917a592d8e","Type":"ContainerStarted","Data":"b715dd81681c59fe9c7d075dbbcf2f6bfd91e95465ef1c32baa7350d26250fbe"} Jul 23 18:23:29.686198 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:29.685763 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34428-64548bc84-gqv2z" Jul 23 18:23:29.712374 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:29.712320 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34428-64548bc84-gqv2z" podStartSLOduration=2.300445292 podStartE2EDuration="2.712305429s" totalImagesPullingTime="411.860137ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:23:29.709505533 +0000 UTC m=+2353.787808886" watchObservedRunningTime="2026-07-23 18:23:29.712305429 +0000 UTC m=+2353.790608782" Jul 23 18:23:35.247999 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:35.247965 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn"] Jul 23 18:23:35.251762 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:35.251742 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn" Jul 23 18:23:35.255708 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:35.255688 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-526dd2-serving-cert\"" Jul 23 18:23:35.290378 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:35.290340 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4544b\" (UniqueName: \"kubernetes.io/projected/a66c0e5c-ae29-4a04-ad89-1e23e0c0f768-kube-api-access-4544b\") pod \"maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn\" (UID: \"a66c0e5c-ae29-4a04-ad89-1e23e0c0f768\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn" Jul 23 18:23:35.290532 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:35.290391 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a66c0e5c-ae29-4a04-ad89-1e23e0c0f768-maas-api-tls\") pod \"maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn\" (UID: \"a66c0e5c-ae29-4a04-ad89-1e23e0c0f768\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn" Jul 23 18:23:35.290532 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:35.290457 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a66c0e5c-ae29-4a04-ad89-1e23e0c0f768-openshift-service-ca\") pod \"maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn\" (UID: \"a66c0e5c-ae29-4a04-ad89-1e23e0c0f768\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn" Jul 23 18:23:35.299102 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:35.299075 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn"] Jul 23 18:23:35.390920 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:35.390887 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4544b\" (UniqueName: \"kubernetes.io/projected/a66c0e5c-ae29-4a04-ad89-1e23e0c0f768-kube-api-access-4544b\") pod \"maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn\" (UID: \"a66c0e5c-ae29-4a04-ad89-1e23e0c0f768\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn" Jul 23 18:23:35.391111 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:35.390940 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a66c0e5c-ae29-4a04-ad89-1e23e0c0f768-maas-api-tls\") pod \"maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn\" (UID: \"a66c0e5c-ae29-4a04-ad89-1e23e0c0f768\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn" Jul 23 18:23:35.391111 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:35.390978 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a66c0e5c-ae29-4a04-ad89-1e23e0c0f768-openshift-service-ca\") pod \"maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn\" (UID: \"a66c0e5c-ae29-4a04-ad89-1e23e0c0f768\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn" Jul 23 18:23:35.391111 ip-10-0-137-82 kubenswrapper[2565]: E0723 18:23:35.391086 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-526dd2-serving-cert: secret "maas-api-e2e-api-b-526dd2-serving-cert" not found Jul 23 18:23:35.391299 ip-10-0-137-82 kubenswrapper[2565]: E0723 18:23:35.391160 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a66c0e5c-ae29-4a04-ad89-1e23e0c0f768-maas-api-tls podName:a66c0e5c-ae29-4a04-ad89-1e23e0c0f768 nodeName:}" failed. No retries permitted until 2026-07-23 18:23:35.891143317 +0000 UTC m=+2359.969446648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a66c0e5c-ae29-4a04-ad89-1e23e0c0f768-maas-api-tls") pod "maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn" (UID: "a66c0e5c-ae29-4a04-ad89-1e23e0c0f768") : secret "maas-api-e2e-api-b-526dd2-serving-cert" not found Jul 23 18:23:35.391643 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:35.391625 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a66c0e5c-ae29-4a04-ad89-1e23e0c0f768-openshift-service-ca\") pod \"maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn\" (UID: \"a66c0e5c-ae29-4a04-ad89-1e23e0c0f768\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn" Jul 23 18:23:35.405211 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:35.405187 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4544b\" (UniqueName: \"kubernetes.io/projected/a66c0e5c-ae29-4a04-ad89-1e23e0c0f768-kube-api-access-4544b\") pod \"maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn\" (UID: \"a66c0e5c-ae29-4a04-ad89-1e23e0c0f768\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn" Jul 23 18:23:35.695612 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:35.695572 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34428-64548bc84-gqv2z" Jul 23 18:23:35.893740 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:35.893705 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a66c0e5c-ae29-4a04-ad89-1e23e0c0f768-maas-api-tls\") pod \"maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn\" (UID: \"a66c0e5c-ae29-4a04-ad89-1e23e0c0f768\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn" Jul 23 18:23:35.896140 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:35.896110 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a66c0e5c-ae29-4a04-ad89-1e23e0c0f768-maas-api-tls\") pod \"maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn\" (UID: \"a66c0e5c-ae29-4a04-ad89-1e23e0c0f768\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn" Jul 23 18:23:36.162481 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:36.162444 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn" Jul 23 18:23:36.324239 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:36.324207 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn"] Jul 23 18:23:36.326644 ip-10-0-137-82 kubenswrapper[2565]: W0723 18:23:36.326611 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda66c0e5c_ae29_4a04_ad89_1e23e0c0f768.slice/crio-11ea2f95d455bf76d515f3b449b6e178fb07e0cf5e796787a83c0f3b099b21c7 WatchSource:0}: Error finding container 11ea2f95d455bf76d515f3b449b6e178fb07e0cf5e796787a83c0f3b099b21c7: Status 404 returned error can't find the container with id 11ea2f95d455bf76d515f3b449b6e178fb07e0cf5e796787a83c0f3b099b21c7 Jul 23 18:23:36.722463 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:36.722424 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn" event={"ID":"a66c0e5c-ae29-4a04-ad89-1e23e0c0f768","Type":"ContainerStarted","Data":"11ea2f95d455bf76d515f3b449b6e178fb07e0cf5e796787a83c0f3b099b21c7"} Jul 23 18:23:37.729352 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:37.729312 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn" event={"ID":"a66c0e5c-ae29-4a04-ad89-1e23e0c0f768","Type":"ContainerStarted","Data":"32253525ceaa9873397b4c308cfab2f418c50ddb84f1f4d37ac5e6d6ed64f4bb"} Jul 23 18:23:37.729793 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:37.729410 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn" Jul 23 18:23:37.753793 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:37.753742 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn" podStartSLOduration=2.25039142 podStartE2EDuration="2.753726979s" totalImagesPullingTime="503.335559ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:23:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:23:37.752362063 +0000 UTC m=+2361.830665415" watchObservedRunningTime="2026-07-23 18:23:37.753726979 +0000 UTC m=+2361.832030332" Jul 23 18:23:43.740286 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:43.740256 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn" Jul 23 18:23:47.543321 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:47.543275 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34428-7ltmz"] Jul 23 18:23:47.547186 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:47.547163 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34428-7ltmz" Jul 23 18:23:47.549884 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:47.549862 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5lmm\"" Jul 23 18:23:47.555231 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:47.555207 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34428-7ltmz"] Jul 23 18:23:47.575021 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:47.574987 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65edfdb8-e460-4020-bac8-3661ae01a388-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b34428-7ltmz\" (UID: \"65edfdb8-e460-4020-bac8-3661ae01a388\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34428-7ltmz" Jul 23 18:23:47.676224 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:47.676180 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65edfdb8-e460-4020-bac8-3661ae01a388-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b34428-7ltmz\" (UID: \"65edfdb8-e460-4020-bac8-3661ae01a388\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34428-7ltmz" Jul 23 18:23:47.676869 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:47.676849 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65edfdb8-e460-4020-bac8-3661ae01a388-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b34428-7ltmz\" (UID: \"65edfdb8-e460-4020-bac8-3661ae01a388\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34428-7ltmz" Jul 23 18:23:47.859647 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:47.859531 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34428-7ltmz" Jul 23 18:23:48.000847 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:48.000823 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34428-7ltmz"] Jul 23 18:23:48.003364 ip-10-0-137-82 kubenswrapper[2565]: W0723 18:23:48.003336 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65edfdb8_e460_4020_bac8_3661ae01a388.slice/crio-e159b85dd5c5218842ab7acee9d40b163ab41eaa49587635d3bd6f8dad89dbf6 WatchSource:0}: Error finding container e159b85dd5c5218842ab7acee9d40b163ab41eaa49587635d3bd6f8dad89dbf6: Status 404 returned error can't find the container with id e159b85dd5c5218842ab7acee9d40b163ab41eaa49587635d3bd6f8dad89dbf6 Jul 23 18:23:48.786629 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:48.786582 2565 generic.go:356] "Generic (PLEG): container finished" podID="65edfdb8-e460-4020-bac8-3661ae01a388" containerID="b5a4380c029d22e67064b92035f6d2a5cc8666af87d84f0c7d82da492220239f" exitCode=0 Jul 23 18:23:48.787001 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:48.786669 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34428-7ltmz" event={"ID":"65edfdb8-e460-4020-bac8-3661ae01a388","Type":"ContainerDied","Data":"b5a4380c029d22e67064b92035f6d2a5cc8666af87d84f0c7d82da492220239f"} Jul 23 18:23:48.787001 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:48.786720 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34428-7ltmz" event={"ID":"65edfdb8-e460-4020-bac8-3661ae01a388","Type":"ContainerStarted","Data":"e159b85dd5c5218842ab7acee9d40b163ab41eaa49587635d3bd6f8dad89dbf6"} Jul 23 18:23:49.934496 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:49.934467 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34428-7ltmz" Jul 23 18:23:49.993924 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:49.993892 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65edfdb8-e460-4020-bac8-3661ae01a388-maas-api-service-ca\") pod \"65edfdb8-e460-4020-bac8-3661ae01a388\" (UID: \"65edfdb8-e460-4020-bac8-3661ae01a388\") " Jul 23 18:23:49.994260 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:49.994232 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65edfdb8-e460-4020-bac8-3661ae01a388-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "65edfdb8-e460-4020-bac8-3661ae01a388" (UID: "65edfdb8-e460-4020-bac8-3661ae01a388"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:23:50.094937 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:50.094837 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65edfdb8-e460-4020-bac8-3661ae01a388-maas-api-service-ca\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:23:50.798432 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:50.798399 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34428-7ltmz" Jul 23 18:23:50.798623 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:50.798448 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34428-7ltmz" event={"ID":"65edfdb8-e460-4020-bac8-3661ae01a388","Type":"ContainerDied","Data":"e159b85dd5c5218842ab7acee9d40b163ab41eaa49587635d3bd6f8dad89dbf6"} Jul 23 18:23:50.798623 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:23:50.798497 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e159b85dd5c5218842ab7acee9d40b163ab41eaa49587635d3bd6f8dad89dbf6" Jul 23 18:24:02.837580 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:02.837543 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b34428-64548bc84-gqv2z"] Jul 23 18:24:02.838118 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:02.837795 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34428-64548bc84-gqv2z" podUID="78f72469-a821-4f59-91b7-25917a592d8e" containerName="maas-api" containerID="cri-o://b715dd81681c59fe9c7d075dbbcf2f6bfd91e95465ef1c32baa7350d26250fbe" gracePeriod=30 Jul 23 18:24:03.086071 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:03.086044 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34428-64548bc84-gqv2z" Jul 23 18:24:03.125958 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:03.125866 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78f72469-a821-4f59-91b7-25917a592d8e-maas-api-tls\") pod \"78f72469-a821-4f59-91b7-25917a592d8e\" (UID: \"78f72469-a821-4f59-91b7-25917a592d8e\") " Jul 23 18:24:03.126098 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:03.125970 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f72469-a821-4f59-91b7-25917a592d8e-openshift-service-ca\") pod \"78f72469-a821-4f59-91b7-25917a592d8e\" (UID: \"78f72469-a821-4f59-91b7-25917a592d8e\") " Jul 23 18:24:03.126098 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:03.126038 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vw68\" (UniqueName: \"kubernetes.io/projected/78f72469-a821-4f59-91b7-25917a592d8e-kube-api-access-8vw68\") pod \"78f72469-a821-4f59-91b7-25917a592d8e\" (UID: \"78f72469-a821-4f59-91b7-25917a592d8e\") " Jul 23 18:24:03.126363 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:03.126332 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78f72469-a821-4f59-91b7-25917a592d8e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "78f72469-a821-4f59-91b7-25917a592d8e" (UID: "78f72469-a821-4f59-91b7-25917a592d8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:24:03.128182 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:03.128153 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78f72469-a821-4f59-91b7-25917a592d8e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "78f72469-a821-4f59-91b7-25917a592d8e" (UID: "78f72469-a821-4f59-91b7-25917a592d8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:24:03.128305 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:03.128284 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78f72469-a821-4f59-91b7-25917a592d8e-kube-api-access-8vw68" (OuterVolumeSpecName: "kube-api-access-8vw68") pod "78f72469-a821-4f59-91b7-25917a592d8e" (UID: "78f72469-a821-4f59-91b7-25917a592d8e"). InnerVolumeSpecName "kube-api-access-8vw68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:24:03.227532 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:03.227500 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78f72469-a821-4f59-91b7-25917a592d8e-maas-api-tls\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:24:03.227532 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:03.227528 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f72469-a821-4f59-91b7-25917a592d8e-openshift-service-ca\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:24:03.227532 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:03.227538 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vw68\" (UniqueName: \"kubernetes.io/projected/78f72469-a821-4f59-91b7-25917a592d8e-kube-api-access-8vw68\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:24:03.865349 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:03.865311 2565 generic.go:356] "Generic (PLEG): container finished" podID="78f72469-a821-4f59-91b7-25917a592d8e" containerID="b715dd81681c59fe9c7d075dbbcf2f6bfd91e95465ef1c32baa7350d26250fbe" exitCode=0 Jul 23 18:24:03.865809 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:03.865356 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34428-64548bc84-gqv2z" Jul 23 18:24:03.865809 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:03.865371 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34428-64548bc84-gqv2z" event={"ID":"78f72469-a821-4f59-91b7-25917a592d8e","Type":"ContainerDied","Data":"b715dd81681c59fe9c7d075dbbcf2f6bfd91e95465ef1c32baa7350d26250fbe"} Jul 23 18:24:03.865809 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:03.865413 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34428-64548bc84-gqv2z" event={"ID":"78f72469-a821-4f59-91b7-25917a592d8e","Type":"ContainerDied","Data":"4315433aa78674e45561beee2dc276378fe8afeadffd096e5eeb7ef8d9e01fa0"} Jul 23 18:24:03.865809 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:03.865436 2565 scope.go:117] "RemoveContainer" containerID="b715dd81681c59fe9c7d075dbbcf2f6bfd91e95465ef1c32baa7350d26250fbe" Jul 23 18:24:03.874574 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:03.874558 2565 scope.go:117] "RemoveContainer" containerID="b715dd81681c59fe9c7d075dbbcf2f6bfd91e95465ef1c32baa7350d26250fbe" Jul 23 18:24:03.874832 ip-10-0-137-82 kubenswrapper[2565]: E0723 18:24:03.874814 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b715dd81681c59fe9c7d075dbbcf2f6bfd91e95465ef1c32baa7350d26250fbe\": container with ID starting with b715dd81681c59fe9c7d075dbbcf2f6bfd91e95465ef1c32baa7350d26250fbe not found: ID does not exist" containerID="b715dd81681c59fe9c7d075dbbcf2f6bfd91e95465ef1c32baa7350d26250fbe" Jul 23 18:24:03.874875 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:03.874841 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b715dd81681c59fe9c7d075dbbcf2f6bfd91e95465ef1c32baa7350d26250fbe"} err="failed to get container status \"b715dd81681c59fe9c7d075dbbcf2f6bfd91e95465ef1c32baa7350d26250fbe\": rpc error: code = NotFound desc = could not find container \"b715dd81681c59fe9c7d075dbbcf2f6bfd91e95465ef1c32baa7350d26250fbe\": container with ID starting with b715dd81681c59fe9c7d075dbbcf2f6bfd91e95465ef1c32baa7350d26250fbe not found: ID does not exist" Jul 23 18:24:03.889656 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:03.889628 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b34428-64548bc84-gqv2z"] Jul 23 18:24:03.891437 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:03.891416 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b34428-64548bc84-gqv2z"] Jul 23 18:24:04.405676 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:04.405644 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78f72469-a821-4f59-91b7-25917a592d8e" path="/var/lib/kubelet/pods/78f72469-a821-4f59-91b7-25917a592d8e/volumes" Jul 23 18:24:07.660605 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:07.660570 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34428-7ltmz"] Jul 23 18:24:07.665272 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:07.665235 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34428-7ltmz"] Jul 23 18:24:08.406229 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:08.406194 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65edfdb8-e460-4020-bac8-3661ae01a388" path="/var/lib/kubelet/pods/65edfdb8-e460-4020-bac8-3661ae01a388/volumes" Jul 23 18:24:14.942891 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:14.942859 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-526dd2-gtz5t"] Jul 23 18:24:14.943340 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:14.943214 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65edfdb8-e460-4020-bac8-3661ae01a388" containerName="revoke-keys" Jul 23 18:24:14.943340 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:14.943225 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="65edfdb8-e460-4020-bac8-3661ae01a388" containerName="revoke-keys" Jul 23 18:24:14.943340 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:14.943243 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78f72469-a821-4f59-91b7-25917a592d8e" containerName="maas-api" Jul 23 18:24:14.943340 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:14.943249 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="78f72469-a821-4f59-91b7-25917a592d8e" containerName="maas-api" Jul 23 18:24:14.943340 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:14.943315 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="65edfdb8-e460-4020-bac8-3661ae01a388" containerName="revoke-keys" Jul 23 18:24:14.943340 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:14.943326 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="78f72469-a821-4f59-91b7-25917a592d8e" containerName="maas-api" Jul 23 18:24:14.948084 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:14.948067 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-526dd2-gtz5t" Jul 23 18:24:14.950919 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:14.950897 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5lmm\"" Jul 23 18:24:14.954082 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:14.954059 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-526dd2-gtz5t"] Jul 23 18:24:15.034740 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:15.034703 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae220007-000b-410a-87bc-84fc3e95bb6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-526dd2-gtz5t\" (UID: \"ae220007-000b-410a-87bc-84fc3e95bb6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-526dd2-gtz5t" Jul 23 18:24:15.135342 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:15.135303 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae220007-000b-410a-87bc-84fc3e95bb6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-526dd2-gtz5t\" (UID: \"ae220007-000b-410a-87bc-84fc3e95bb6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-526dd2-gtz5t" Jul 23 18:24:15.136039 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:15.136011 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae220007-000b-410a-87bc-84fc3e95bb6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-526dd2-gtz5t\" (UID: \"ae220007-000b-410a-87bc-84fc3e95bb6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-526dd2-gtz5t" Jul 23 18:24:15.261513 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:15.261420 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-526dd2-gtz5t" Jul 23 18:24:15.398147 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:15.398120 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-526dd2-gtz5t"] Jul 23 18:24:15.400006 ip-10-0-137-82 kubenswrapper[2565]: W0723 18:24:15.399977 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae220007_000b_410a_87bc_84fc3e95bb6f.slice/crio-b57031de7c0a5835563eec54dde3e8c824a521642dc4d3edf8f6104a47ff82d2 WatchSource:0}: Error finding container b57031de7c0a5835563eec54dde3e8c824a521642dc4d3edf8f6104a47ff82d2: Status 404 returned error can't find the container with id b57031de7c0a5835563eec54dde3e8c824a521642dc4d3edf8f6104a47ff82d2 Jul 23 18:24:15.931458 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:15.931361 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-526dd2-gtz5t" event={"ID":"ae220007-000b-410a-87bc-84fc3e95bb6f","Type":"ContainerStarted","Data":"0b7e4f2b98979776a9c920eb6e9b861c880cdf836f338d611589a949c1aee0dd"} Jul 23 18:24:15.931458 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:15.931428 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-526dd2-gtz5t" event={"ID":"ae220007-000b-410a-87bc-84fc3e95bb6f","Type":"ContainerStarted","Data":"b57031de7c0a5835563eec54dde3e8c824a521642dc4d3edf8f6104a47ff82d2"} Jul 23 18:24:15.946764 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:15.946715 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-526dd2-gtz5t" podStartSLOduration=1.9466963480000001 podStartE2EDuration="1.946696348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:24:15.946391002 +0000 UTC m=+2400.024694359" watchObservedRunningTime="2026-07-23 18:24:15.946696348 +0000 UTC m=+2400.024999701" Jul 23 18:24:16.517062 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:16.517026 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-m7k4j_1cca3236-5c90-4abb-9b9b-39e40b1903c3/console-operator/1.log" Jul 23 18:24:16.524547 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:16.524523 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-m7k4j_1cca3236-5c90-4abb-9b9b-39e40b1903c3/console-operator/1.log" Jul 23 18:24:16.663083 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:16.663052 2565 scope.go:117] "RemoveContainer" containerID="4de6ad418321750e51b204289ab35680e818d0fcb1da7ba94c232dc68d7f717f" Jul 23 18:24:16.937394 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:16.937302 2565 generic.go:356] "Generic (PLEG): container finished" podID="ae220007-000b-410a-87bc-84fc3e95bb6f" containerID="0b7e4f2b98979776a9c920eb6e9b861c880cdf836f338d611589a949c1aee0dd" exitCode=0 Jul 23 18:24:16.937394 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:16.937379 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-526dd2-gtz5t" event={"ID":"ae220007-000b-410a-87bc-84fc3e95bb6f","Type":"ContainerDied","Data":"0b7e4f2b98979776a9c920eb6e9b861c880cdf836f338d611589a949c1aee0dd"} Jul 23 18:24:18.083868 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:18.083844 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-526dd2-gtz5t" Jul 23 18:24:18.165111 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:18.165073 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae220007-000b-410a-87bc-84fc3e95bb6f-maas-api-service-ca\") pod \"ae220007-000b-410a-87bc-84fc3e95bb6f\" (UID: \"ae220007-000b-410a-87bc-84fc3e95bb6f\") " Jul 23 18:24:18.165486 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:18.165462 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae220007-000b-410a-87bc-84fc3e95bb6f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ae220007-000b-410a-87bc-84fc3e95bb6f" (UID: "ae220007-000b-410a-87bc-84fc3e95bb6f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:24:18.266168 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:18.266118 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae220007-000b-410a-87bc-84fc3e95bb6f-maas-api-service-ca\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:24:18.948365 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:18.948331 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-526dd2-gtz5t" event={"ID":"ae220007-000b-410a-87bc-84fc3e95bb6f","Type":"ContainerDied","Data":"b57031de7c0a5835563eec54dde3e8c824a521642dc4d3edf8f6104a47ff82d2"} Jul 23 18:24:18.948365 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:18.948365 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-526dd2-gtz5t" Jul 23 18:24:18.948579 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:18.948381 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b57031de7c0a5835563eec54dde3e8c824a521642dc4d3edf8f6104a47ff82d2" Jul 23 18:24:30.787045 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:30.787008 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn"] Jul 23 18:24:30.787420 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:30.787255 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn" podUID="a66c0e5c-ae29-4a04-ad89-1e23e0c0f768" containerName="maas-api" containerID="cri-o://32253525ceaa9873397b4c308cfab2f418c50ddb84f1f4d37ac5e6d6ed64f4bb" gracePeriod=30 Jul 23 18:24:31.010258 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:31.010223 2565 generic.go:356] "Generic (PLEG): container finished" podID="a66c0e5c-ae29-4a04-ad89-1e23e0c0f768" containerID="32253525ceaa9873397b4c308cfab2f418c50ddb84f1f4d37ac5e6d6ed64f4bb" exitCode=0 Jul 23 18:24:31.010456 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:31.010267 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn" event={"ID":"a66c0e5c-ae29-4a04-ad89-1e23e0c0f768","Type":"ContainerDied","Data":"32253525ceaa9873397b4c308cfab2f418c50ddb84f1f4d37ac5e6d6ed64f4bb"} Jul 23 18:24:31.042210 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:31.042150 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn" Jul 23 18:24:31.081038 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:31.080959 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a66c0e5c-ae29-4a04-ad89-1e23e0c0f768-openshift-service-ca\") pod \"a66c0e5c-ae29-4a04-ad89-1e23e0c0f768\" (UID: \"a66c0e5c-ae29-4a04-ad89-1e23e0c0f768\") " Jul 23 18:24:31.081189 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:31.081121 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4544b\" (UniqueName: \"kubernetes.io/projected/a66c0e5c-ae29-4a04-ad89-1e23e0c0f768-kube-api-access-4544b\") pod \"a66c0e5c-ae29-4a04-ad89-1e23e0c0f768\" (UID: \"a66c0e5c-ae29-4a04-ad89-1e23e0c0f768\") " Jul 23 18:24:31.081189 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:31.081148 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a66c0e5c-ae29-4a04-ad89-1e23e0c0f768-maas-api-tls\") pod \"a66c0e5c-ae29-4a04-ad89-1e23e0c0f768\" (UID: \"a66c0e5c-ae29-4a04-ad89-1e23e0c0f768\") " Jul 23 18:24:31.081317 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:31.081237 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a66c0e5c-ae29-4a04-ad89-1e23e0c0f768-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a66c0e5c-ae29-4a04-ad89-1e23e0c0f768" (UID: "a66c0e5c-ae29-4a04-ad89-1e23e0c0f768"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:24:31.081421 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:31.081397 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a66c0e5c-ae29-4a04-ad89-1e23e0c0f768-openshift-service-ca\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:24:31.083290 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:31.083259 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a66c0e5c-ae29-4a04-ad89-1e23e0c0f768-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a66c0e5c-ae29-4a04-ad89-1e23e0c0f768" (UID: "a66c0e5c-ae29-4a04-ad89-1e23e0c0f768"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:24:31.083418 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:31.083371 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a66c0e5c-ae29-4a04-ad89-1e23e0c0f768-kube-api-access-4544b" (OuterVolumeSpecName: "kube-api-access-4544b") pod "a66c0e5c-ae29-4a04-ad89-1e23e0c0f768" (UID: "a66c0e5c-ae29-4a04-ad89-1e23e0c0f768"). InnerVolumeSpecName "kube-api-access-4544b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:24:31.182621 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:31.182565 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4544b\" (UniqueName: \"kubernetes.io/projected/a66c0e5c-ae29-4a04-ad89-1e23e0c0f768-kube-api-access-4544b\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:24:31.182621 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:31.182616 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a66c0e5c-ae29-4a04-ad89-1e23e0c0f768-maas-api-tls\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:24:32.016083 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:32.016053 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn" Jul 23 18:24:32.016489 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:32.016095 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn" event={"ID":"a66c0e5c-ae29-4a04-ad89-1e23e0c0f768","Type":"ContainerDied","Data":"11ea2f95d455bf76d515f3b449b6e178fb07e0cf5e796787a83c0f3b099b21c7"} Jul 23 18:24:32.016489 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:32.016161 2565 scope.go:117] "RemoveContainer" containerID="32253525ceaa9873397b4c308cfab2f418c50ddb84f1f4d37ac5e6d6ed64f4bb" Jul 23 18:24:32.055804 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:32.055774 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn"] Jul 23 18:24:32.064230 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:32.064202 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-526dd2-dcf686b9c-vmwhn"] Jul 23 18:24:32.405444 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:32.405349 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a66c0e5c-ae29-4a04-ad89-1e23e0c0f768" path="/var/lib/kubelet/pods/a66c0e5c-ae29-4a04-ad89-1e23e0c0f768/volumes" Jul 23 18:24:35.111578 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:35.111532 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-526dd2-gtz5t"] Jul 23 18:24:35.114726 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:35.114697 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-526dd2-gtz5t"] Jul 23 18:24:36.405742 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:36.405703 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae220007-000b-410a-87bc-84fc3e95bb6f" path="/var/lib/kubelet/pods/ae220007-000b-410a-87bc-84fc3e95bb6f/volumes" Jul 23 18:24:46.513871 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:46.513784 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp"] Jul 23 18:24:46.514349 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:46.514333 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae220007-000b-410a-87bc-84fc3e95bb6f" containerName="revoke-keys" Jul 23 18:24:46.514404 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:46.514353 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae220007-000b-410a-87bc-84fc3e95bb6f" containerName="revoke-keys" Jul 23 18:24:46.514404 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:46.514382 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a66c0e5c-ae29-4a04-ad89-1e23e0c0f768" containerName="maas-api" Jul 23 18:24:46.514404 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:46.514391 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a66c0e5c-ae29-4a04-ad89-1e23e0c0f768" containerName="maas-api" Jul 23 18:24:46.514511 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:46.514473 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae220007-000b-410a-87bc-84fc3e95bb6f" containerName="revoke-keys" Jul 23 18:24:46.514511 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:46.514483 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a66c0e5c-ae29-4a04-ad89-1e23e0c0f768" containerName="maas-api" Jul 23 18:24:46.519316 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:46.519269 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp" Jul 23 18:24:46.522031 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:46.521991 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-4d95fb-serving-cert\"" Jul 23 18:24:46.527880 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:46.527857 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp"] Jul 23 18:24:46.619983 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:46.619946 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2-maas-api-tls\") pod \"maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp\" (UID: \"ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp" Jul 23 18:24:46.620190 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:46.620001 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qngm7\" (UniqueName: \"kubernetes.io/projected/ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2-kube-api-access-qngm7\") pod \"maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp\" (UID: \"ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp" Jul 23 18:24:46.620190 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:46.620095 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp\" (UID: \"ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp" Jul 23 18:24:46.721058 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:46.721024 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2-maas-api-tls\") pod \"maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp\" (UID: \"ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp" Jul 23 18:24:46.721235 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:46.721074 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qngm7\" (UniqueName: \"kubernetes.io/projected/ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2-kube-api-access-qngm7\") pod \"maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp\" (UID: \"ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp" Jul 23 18:24:46.721235 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:46.721121 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp\" (UID: \"ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp" Jul 23 18:24:46.721908 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:46.721883 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp\" (UID: \"ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp" Jul 23 18:24:46.723755 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:46.723730 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2-maas-api-tls\") pod \"maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp\" (UID: \"ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp" Jul 23 18:24:46.747548 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:46.747514 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qngm7\" (UniqueName: \"kubernetes.io/projected/ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2-kube-api-access-qngm7\") pod \"maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp\" (UID: \"ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp" Jul 23 18:24:46.832915 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:46.832832 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp" Jul 23 18:24:47.015142 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:47.015116 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp"] Jul 23 18:24:47.017699 ip-10-0-137-82 kubenswrapper[2565]: W0723 18:24:47.017668 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae32480f_8ce0_4ace_a3f8_38ccb2d9ddb2.slice/crio-d3f950fdd1096714715842bc805b46e05be600273dab4180e52456a1c2e4d246 WatchSource:0}: Error finding container d3f950fdd1096714715842bc805b46e05be600273dab4180e52456a1c2e4d246: Status 404 returned error can't find the container with id d3f950fdd1096714715842bc805b46e05be600273dab4180e52456a1c2e4d246 Jul 23 18:24:47.094355 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:47.094266 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp" event={"ID":"ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2","Type":"ContainerStarted","Data":"d3f950fdd1096714715842bc805b46e05be600273dab4180e52456a1c2e4d246"} Jul 23 18:24:48.101818 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:48.101768 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp" event={"ID":"ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2","Type":"ContainerStarted","Data":"ccf826b5a0411f6a37bb61c74c05f18849e60a5fcb1b25ee6d3d384ae86af16a"} Jul 23 18:24:48.102204 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:48.101853 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp" Jul 23 18:24:48.123965 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:48.123911 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp" podStartSLOduration=1.546356531 podStartE2EDuration="2.123896829s" totalImagesPullingTime="577.540298ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:24:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:24:48.122629193 +0000 UTC m=+2432.200932557" watchObservedRunningTime="2026-07-23 18:24:48.123896829 +0000 UTC m=+2432.202200181" Jul 23 18:24:54.113554 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:54.113525 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp" Jul 23 18:24:54.601857 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:54.601821 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw"] Jul 23 18:24:54.605691 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:54.605672 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw" Jul 23 18:24:54.609218 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:54.609187 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-e1d86b-serving-cert\"" Jul 23 18:24:54.618948 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:54.618912 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw"] Jul 23 18:24:54.689519 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:54.689488 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ce25c79-d0a8-46f1-9008-d236136d56f0-maas-api-tls\") pod \"maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw\" (UID: \"0ce25c79-d0a8-46f1-9008-d236136d56f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw" Jul 23 18:24:54.689707 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:54.689549 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce25c79-d0a8-46f1-9008-d236136d56f0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw\" (UID: \"0ce25c79-d0a8-46f1-9008-d236136d56f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw" Jul 23 18:24:54.689707 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:54.689617 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnc62\" (UniqueName: \"kubernetes.io/projected/0ce25c79-d0a8-46f1-9008-d236136d56f0-kube-api-access-fnc62\") pod \"maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw\" (UID: \"0ce25c79-d0a8-46f1-9008-d236136d56f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw" Jul 23 18:24:54.790799 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:54.790758 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ce25c79-d0a8-46f1-9008-d236136d56f0-maas-api-tls\") pod \"maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw\" (UID: \"0ce25c79-d0a8-46f1-9008-d236136d56f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw" Jul 23 18:24:54.790966 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:54.790834 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce25c79-d0a8-46f1-9008-d236136d56f0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw\" (UID: \"0ce25c79-d0a8-46f1-9008-d236136d56f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw" Jul 23 18:24:54.790966 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:54.790882 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnc62\" (UniqueName: \"kubernetes.io/projected/0ce25c79-d0a8-46f1-9008-d236136d56f0-kube-api-access-fnc62\") pod \"maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw\" (UID: \"0ce25c79-d0a8-46f1-9008-d236136d56f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw" Jul 23 18:24:54.791068 ip-10-0-137-82 kubenswrapper[2565]: E0723 18:24:54.791043 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-e1d86b-serving-cert: secret "maas-api-e2e-shared-b-e1d86b-serving-cert" not found Jul 23 18:24:54.791131 ip-10-0-137-82 kubenswrapper[2565]: E0723 18:24:54.791119 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ce25c79-d0a8-46f1-9008-d236136d56f0-maas-api-tls podName:0ce25c79-d0a8-46f1-9008-d236136d56f0 nodeName:}" failed. No retries permitted until 2026-07-23 18:24:55.291098407 +0000 UTC m=+2439.369401742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0ce25c79-d0a8-46f1-9008-d236136d56f0-maas-api-tls") pod "maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw" (UID: "0ce25c79-d0a8-46f1-9008-d236136d56f0") : secret "maas-api-e2e-shared-b-e1d86b-serving-cert" not found Jul 23 18:24:54.791516 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:54.791498 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce25c79-d0a8-46f1-9008-d236136d56f0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw\" (UID: \"0ce25c79-d0a8-46f1-9008-d236136d56f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw" Jul 23 18:24:54.799894 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:54.799861 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnc62\" (UniqueName: \"kubernetes.io/projected/0ce25c79-d0a8-46f1-9008-d236136d56f0-kube-api-access-fnc62\") pod \"maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw\" (UID: \"0ce25c79-d0a8-46f1-9008-d236136d56f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw" Jul 23 18:24:55.295495 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:55.295461 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ce25c79-d0a8-46f1-9008-d236136d56f0-maas-api-tls\") pod \"maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw\" (UID: \"0ce25c79-d0a8-46f1-9008-d236136d56f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw" Jul 23 18:24:55.297882 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:55.297851 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ce25c79-d0a8-46f1-9008-d236136d56f0-maas-api-tls\") pod \"maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw\" (UID: \"0ce25c79-d0a8-46f1-9008-d236136d56f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw" Jul 23 18:24:55.519524 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:55.519488 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw" Jul 23 18:24:55.663921 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:55.663893 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw"] Jul 23 18:24:55.666341 ip-10-0-137-82 kubenswrapper[2565]: W0723 18:24:55.666300 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ce25c79_d0a8_46f1_9008_d236136d56f0.slice/crio-7797dc89e49f0b751606d6e488867f2f78bf1de71cd5a51d110a337c122273b5 WatchSource:0}: Error finding container 7797dc89e49f0b751606d6e488867f2f78bf1de71cd5a51d110a337c122273b5: Status 404 returned error can't find the container with id 7797dc89e49f0b751606d6e488867f2f78bf1de71cd5a51d110a337c122273b5 Jul 23 18:24:56.145834 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:56.145794 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw" event={"ID":"0ce25c79-d0a8-46f1-9008-d236136d56f0","Type":"ContainerStarted","Data":"e9845637a476ec6867847f9ecbbc0f8e72fb8a83ce12ec731c0ebae31e918d6b"} Jul 23 18:24:56.146017 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:56.145851 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw" event={"ID":"0ce25c79-d0a8-46f1-9008-d236136d56f0","Type":"ContainerStarted","Data":"7797dc89e49f0b751606d6e488867f2f78bf1de71cd5a51d110a337c122273b5"} Jul 23 18:24:56.146017 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:56.145874 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw" Jul 23 18:24:56.173094 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:24:56.173043 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw" podStartSLOduration=1.77752921 podStartE2EDuration="2.173027474s" totalImagesPullingTime="395.498264ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:24:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:24:56.170269379 +0000 UTC m=+2440.248572732" watchObservedRunningTime="2026-07-23 18:24:56.173027474 +0000 UTC m=+2440.251330826" Jul 23 18:25:02.156657 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:25:02.156616 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw" Jul 23 18:25:16.688775 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:25:16.688741 2565 scope.go:117] "RemoveContainer" containerID="69054b41636b31e85d11e6082a87ea2a7cd3bcbe8f0745f8a85b7e0a7a0b6f65" Jul 23 18:29:16.546405 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:29:16.546328 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-m7k4j_1cca3236-5c90-4abb-9b9b-39e40b1903c3/console-operator/1.log" Jul 23 18:29:16.556158 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:29:16.556133 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-m7k4j_1cca3236-5c90-4abb-9b9b-39e40b1903c3/console-operator/1.log" Jul 23 18:29:16.737174 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:29:16.737144 2565 scope.go:117] "RemoveContainer" containerID="4833380c49c39ae6c21ccbfdb5b973f4a4f5ff8539784fcb2891678f20e36b81" Jul 23 18:29:16.745668 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:29:16.745647 2565 scope.go:117] "RemoveContainer" containerID="45178ba2c28966632b4abc900c65b2fed0960b881604f12546e41ad2c30bafd5" Jul 23 18:30:00.137908 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:00.137866 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4d95fb-29747190-tz4x9"] Jul 23 18:30:00.141185 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:00.141166 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e1d86b-29747190-hscnb"] Jul 23 18:30:00.141303 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:00.141281 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4d95fb-29747190-tz4x9" Jul 23 18:30:00.144123 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:00.144096 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5lmm\"" Jul 23 18:30:00.144483 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:00.144468 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747190-kc7kw"] Jul 23 18:30:00.144580 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:00.144567 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e1d86b-29747190-hscnb" Jul 23 18:30:00.147893 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:00.147875 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747190-kc7kw" Jul 23 18:30:00.152829 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:00.152798 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4d95fb-29747190-tz4x9"] Jul 23 18:30:00.154958 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:00.154934 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4d95fb-29747190-tz4x9" Jul 23 18:30:00.155455 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:00.155435 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e1d86b-29747190-hscnb"] Jul 23 18:30:00.165746 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:00.165721 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e1d86b-29747190-hscnb" Jul 23 18:30:00.171195 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:00.171170 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747190-kc7kw"] Jul 23 18:30:00.206563 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:00.206538 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747190-kc7kw" Jul 23 18:30:00.407559 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:00.407516 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747190-kc7kw"] Jul 23 18:30:00.408752 ip-10-0-137-82 kubenswrapper[2565]: W0723 18:30:00.408729 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8bab927_792a_4fbe_8d53_522524b92670.slice/crio-520c68e10b2917d5e3b29d13e6a732eda732614c18924b4297c8dffcc0ff1b06 WatchSource:0}: Error finding container 520c68e10b2917d5e3b29d13e6a732eda732614c18924b4297c8dffcc0ff1b06: Status 404 returned error can't find the container with id 520c68e10b2917d5e3b29d13e6a732eda732614c18924b4297c8dffcc0ff1b06 Jul 23 18:30:00.538710 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:00.538642 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4d95fb-29747190-tz4x9"] Jul 23 18:30:00.541033 ip-10-0-137-82 kubenswrapper[2565]: W0723 18:30:00.541008 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8d45bea_1e73_4c4a_87d9_b23711bd80dd.slice/crio-05a5dec2c22f937d7290d9645b27814d770cadb98eba13adcdbc6f73410562e1 WatchSource:0}: Error finding container 05a5dec2c22f937d7290d9645b27814d770cadb98eba13adcdbc6f73410562e1: Status 404 returned error can't find the container with id 05a5dec2c22f937d7290d9645b27814d770cadb98eba13adcdbc6f73410562e1 Jul 23 18:30:00.580087 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:00.580059 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e1d86b-29747190-hscnb"] Jul 23 18:30:00.580882 ip-10-0-137-82 kubenswrapper[2565]: W0723 18:30:00.580856 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3998af9_9811_4f51_84dd_2ef45e51d626.slice/crio-0dc0b7b2d9e3e116f1b4af2f3120c0e884859ea3a076a4c4423d1d4a1e52c8f8 WatchSource:0}: Error finding container 0dc0b7b2d9e3e116f1b4af2f3120c0e884859ea3a076a4c4423d1d4a1e52c8f8: Status 404 returned error can't find the container with id 0dc0b7b2d9e3e116f1b4af2f3120c0e884859ea3a076a4c4423d1d4a1e52c8f8 Jul 23 18:30:00.693947 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:00.693856 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e1d86b-29747190-hscnb" event={"ID":"c3998af9-9811-4f51-84dd-2ef45e51d626","Type":"ContainerStarted","Data":"0dc0b7b2d9e3e116f1b4af2f3120c0e884859ea3a076a4c4423d1d4a1e52c8f8"} Jul 23 18:30:00.695082 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:00.695060 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747190-kc7kw" event={"ID":"c8bab927-792a-4fbe-8d53-522524b92670","Type":"ContainerStarted","Data":"520c68e10b2917d5e3b29d13e6a732eda732614c18924b4297c8dffcc0ff1b06"} Jul 23 18:30:00.696221 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:00.696197 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4d95fb-29747190-tz4x9" event={"ID":"d8d45bea-1e73-4c4a-87d9-b23711bd80dd","Type":"ContainerStarted","Data":"05a5dec2c22f937d7290d9645b27814d770cadb98eba13adcdbc6f73410562e1"} Jul 23 18:30:01.703317 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:01.703282 2565 generic.go:356] "Generic (PLEG): container finished" podID="c8bab927-792a-4fbe-8d53-522524b92670" containerID="b4a7caafff0700f043e445eb733eaf90a2e03da889145b52761f66dcf273fda0" exitCode=0 Jul 23 18:30:01.703740 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:01.703360 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747190-kc7kw" event={"ID":"c8bab927-792a-4fbe-8d53-522524b92670","Type":"ContainerDied","Data":"b4a7caafff0700f043e445eb733eaf90a2e03da889145b52761f66dcf273fda0"} Jul 23 18:30:01.704917 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:01.704896 2565 generic.go:356] "Generic (PLEG): container finished" podID="d8d45bea-1e73-4c4a-87d9-b23711bd80dd" containerID="660962343c75c9bd2cc34ff2388312b51614dbb4ae5c1876cf1b7c65e6648de7" exitCode=0 Jul 23 18:30:01.705010 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:01.704970 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4d95fb-29747190-tz4x9" event={"ID":"d8d45bea-1e73-4c4a-87d9-b23711bd80dd","Type":"ContainerDied","Data":"660962343c75c9bd2cc34ff2388312b51614dbb4ae5c1876cf1b7c65e6648de7"} Jul 23 18:30:01.706264 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:01.706241 2565 generic.go:356] "Generic (PLEG): container finished" podID="c3998af9-9811-4f51-84dd-2ef45e51d626" containerID="5b9200a8b368ee05dc9a6c1e6a283345f3d396593e7bb1aa1816bfddced2776c" exitCode=0 Jul 23 18:30:01.706346 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:01.706294 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e1d86b-29747190-hscnb" event={"ID":"c3998af9-9811-4f51-84dd-2ef45e51d626","Type":"ContainerDied","Data":"5b9200a8b368ee05dc9a6c1e6a283345f3d396593e7bb1aa1816bfddced2776c"} Jul 23 18:30:02.860532 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:02.860501 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4d95fb-29747190-tz4x9" Jul 23 18:30:02.889877 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:02.889848 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e1d86b-29747190-hscnb" Jul 23 18:30:02.944391 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:02.944356 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747190-kc7kw" Jul 23 18:30:03.717556 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:03.717529 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4d95fb-29747190-tz4x9" Jul 23 18:30:03.717556 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:03.717547 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4d95fb-29747190-tz4x9" event={"ID":"d8d45bea-1e73-4c4a-87d9-b23711bd80dd","Type":"ContainerDied","Data":"05a5dec2c22f937d7290d9645b27814d770cadb98eba13adcdbc6f73410562e1"} Jul 23 18:30:03.717844 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:03.717589 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05a5dec2c22f937d7290d9645b27814d770cadb98eba13adcdbc6f73410562e1" Jul 23 18:30:03.718943 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:03.718922 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e1d86b-29747190-hscnb" Jul 23 18:30:03.719076 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:03.718968 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e1d86b-29747190-hscnb" event={"ID":"c3998af9-9811-4f51-84dd-2ef45e51d626","Type":"ContainerDied","Data":"0dc0b7b2d9e3e116f1b4af2f3120c0e884859ea3a076a4c4423d1d4a1e52c8f8"} Jul 23 18:30:03.719076 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:03.719011 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0dc0b7b2d9e3e116f1b4af2f3120c0e884859ea3a076a4c4423d1d4a1e52c8f8" Jul 23 18:30:03.720679 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:03.720653 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747190-kc7kw" event={"ID":"c8bab927-792a-4fbe-8d53-522524b92670","Type":"ContainerDied","Data":"520c68e10b2917d5e3b29d13e6a732eda732614c18924b4297c8dffcc0ff1b06"} Jul 23 18:30:03.720813 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:03.720692 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747190-kc7kw" Jul 23 18:30:03.720813 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:03.720701 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="520c68e10b2917d5e3b29d13e6a732eda732614c18924b4297c8dffcc0ff1b06" Jul 23 18:30:16.768032 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:16.768002 2565 scope.go:117] "RemoveContainer" containerID="b5a4380c029d22e67064b92035f6d2a5cc8666af87d84f0c7d82da492220239f" Jul 23 18:30:16.776353 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:16.776327 2565 scope.go:117] "RemoveContainer" containerID="0b7e4f2b98979776a9c920eb6e9b861c880cdf836f338d611589a949c1aee0dd" Jul 23 18:30:27.962201 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:27.962157 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7415-140008656601136"] Jul 23 18:30:27.962839 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:27.962723 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8d45bea-1e73-4c4a-87d9-b23711bd80dd" containerName="cleanup" Jul 23 18:30:27.962839 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:27.962746 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8d45bea-1e73-4c4a-87d9-b23711bd80dd" containerName="cleanup" Jul 23 18:30:27.962839 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:27.962769 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3998af9-9811-4f51-84dd-2ef45e51d626" containerName="cleanup" Jul 23 18:30:27.962839 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:27.962778 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3998af9-9811-4f51-84dd-2ef45e51d626" containerName="cleanup" Jul 23 18:30:27.962839 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:27.962820 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8bab927-792a-4fbe-8d53-522524b92670" containerName="cleanup" Jul 23 18:30:27.962839 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:27.962828 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8bab927-792a-4fbe-8d53-522524b92670" containerName="cleanup" Jul 23 18:30:27.963166 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:27.962933 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3998af9-9811-4f51-84dd-2ef45e51d626" containerName="cleanup" Jul 23 18:30:27.963166 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:27.962949 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8d45bea-1e73-4c4a-87d9-b23711bd80dd" containerName="cleanup" Jul 23 18:30:27.963166 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:27.962962 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8bab927-792a-4fbe-8d53-522524b92670" containerName="cleanup" Jul 23 18:30:27.966096 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:27.966074 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-140008656601136" Jul 23 18:30:27.968731 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:27.968713 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-r99xg\"" Jul 23 18:30:27.972467 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:27.972447 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7415-140008656601136"] Jul 23 18:30:28.139946 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:28.139911 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqq4c\" (UniqueName: \"kubernetes.io/projected/03731689-9ad9-4aea-8329-67fc05b8b51e-kube-api-access-hqq4c\") pod \"test-curl-7415-140008656601136\" (UID: \"03731689-9ad9-4aea-8329-67fc05b8b51e\") " pod="opendatahub/test-curl-7415-140008656601136" Jul 23 18:30:28.241140 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:28.241030 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqq4c\" (UniqueName: \"kubernetes.io/projected/03731689-9ad9-4aea-8329-67fc05b8b51e-kube-api-access-hqq4c\") pod \"test-curl-7415-140008656601136\" (UID: \"03731689-9ad9-4aea-8329-67fc05b8b51e\") " pod="opendatahub/test-curl-7415-140008656601136" Jul 23 18:30:28.248850 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:28.248821 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqq4c\" (UniqueName: \"kubernetes.io/projected/03731689-9ad9-4aea-8329-67fc05b8b51e-kube-api-access-hqq4c\") pod \"test-curl-7415-140008656601136\" (UID: \"03731689-9ad9-4aea-8329-67fc05b8b51e\") " pod="opendatahub/test-curl-7415-140008656601136" Jul 23 18:30:28.276875 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:28.276853 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-140008656601136" Jul 23 18:30:28.409553 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:28.409522 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7415-140008656601136"] Jul 23 18:30:28.411206 ip-10-0-137-82 kubenswrapper[2565]: W0723 18:30:28.411179 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03731689_9ad9_4aea_8329_67fc05b8b51e.slice/crio-8bcd667571cddb75ba01d22ba9559d0a4bec11e7e123dbc3add9231315dbe375 WatchSource:0}: Error finding container 8bcd667571cddb75ba01d22ba9559d0a4bec11e7e123dbc3add9231315dbe375: Status 404 returned error can't find the container with id 8bcd667571cddb75ba01d22ba9559d0a4bec11e7e123dbc3add9231315dbe375 Jul 23 18:30:28.412411 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:28.412396 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 18:30:28.848670 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:28.848630 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7415-140008656601136" event={"ID":"03731689-9ad9-4aea-8329-67fc05b8b51e","Type":"ContainerStarted","Data":"8bcd667571cddb75ba01d22ba9559d0a4bec11e7e123dbc3add9231315dbe375"} Jul 23 18:30:29.854744 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:29.854706 2565 generic.go:356] "Generic (PLEG): container finished" podID="03731689-9ad9-4aea-8329-67fc05b8b51e" containerID="5f125e21494306475d657ae790d02bd1084e69b3edd3977bda9ade18f1238958" exitCode=0 Jul 23 18:30:29.855148 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:29.854794 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7415-140008656601136" event={"ID":"03731689-9ad9-4aea-8329-67fc05b8b51e","Type":"ContainerDied","Data":"5f125e21494306475d657ae790d02bd1084e69b3edd3977bda9ade18f1238958"} Jul 23 18:30:30.995241 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:30.995217 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-140008656601136" Jul 23 18:30:31.062340 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:31.062312 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7415-140008656601136_03731689-9ad9-4aea-8329-67fc05b8b51e/test-curl-7415-140008656601136/0.log" Jul 23 18:30:31.092082 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:31.092047 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7415-140008656601136"] Jul 23 18:30:31.097828 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:31.097801 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7415-140008656601136"] Jul 23 18:30:31.170508 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:31.170421 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqq4c\" (UniqueName: \"kubernetes.io/projected/03731689-9ad9-4aea-8329-67fc05b8b51e-kube-api-access-hqq4c\") pod \"03731689-9ad9-4aea-8329-67fc05b8b51e\" (UID: \"03731689-9ad9-4aea-8329-67fc05b8b51e\") " Jul 23 18:30:31.172665 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:31.172632 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03731689-9ad9-4aea-8329-67fc05b8b51e-kube-api-access-hqq4c" (OuterVolumeSpecName: "kube-api-access-hqq4c") pod "03731689-9ad9-4aea-8329-67fc05b8b51e" (UID: "03731689-9ad9-4aea-8329-67fc05b8b51e"). InnerVolumeSpecName "kube-api-access-hqq4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:30:31.271738 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:31.271704 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqq4c\" (UniqueName: \"kubernetes.io/projected/03731689-9ad9-4aea-8329-67fc05b8b51e-kube-api-access-hqq4c\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:30:31.865820 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:31.865788 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-140008656601136" Jul 23 18:30:31.866021 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:31.865796 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bcd667571cddb75ba01d22ba9559d0a4bec11e7e123dbc3add9231315dbe375" Jul 23 18:30:31.876414 ip-10-0-137-82 kubenswrapper[2565]: E0723 18:30:31.876381 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7415-140008656601136\" is forbidden: User \"system:node:ip-10-0-137-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-82.ec2.internal' and this object" podUID="03731689-9ad9-4aea-8329-67fc05b8b51e" pod="opendatahub/test-curl-7415-140008656601136" Jul 23 18:30:32.405698 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:32.405664 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03731689-9ad9-4aea-8329-67fc05b8b51e" path="/var/lib/kubelet/pods/03731689-9ad9-4aea-8329-67fc05b8b51e/volumes" Jul 23 18:30:40.321853 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:40.321760 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7415-140008656553424"] Jul 23 18:30:40.322306 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:40.322288 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03731689-9ad9-4aea-8329-67fc05b8b51e" containerName="test-curl-7415-140008656601136" Jul 23 18:30:40.322360 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:40.322308 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="03731689-9ad9-4aea-8329-67fc05b8b51e" containerName="test-curl-7415-140008656601136" Jul 23 18:30:40.322423 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:40.322412 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="03731689-9ad9-4aea-8329-67fc05b8b51e" containerName="test-curl-7415-140008656601136" Jul 23 18:30:40.326024 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:40.326006 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-140008656553424" Jul 23 18:30:40.328439 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:40.328414 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-r99xg\"" Jul 23 18:30:40.331261 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:40.331234 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7415-140008656553424"] Jul 23 18:30:40.347913 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:40.347885 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdbcs\" (UniqueName: \"kubernetes.io/projected/eebcdeb3-6f61-453c-b6a8-cd1b546d6d51-kube-api-access-tdbcs\") pod \"test-curl-7415-140008656553424\" (UID: \"eebcdeb3-6f61-453c-b6a8-cd1b546d6d51\") " pod="opendatahub/test-curl-7415-140008656553424" Jul 23 18:30:40.448529 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:40.448497 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdbcs\" (UniqueName: \"kubernetes.io/projected/eebcdeb3-6f61-453c-b6a8-cd1b546d6d51-kube-api-access-tdbcs\") pod \"test-curl-7415-140008656553424\" (UID: \"eebcdeb3-6f61-453c-b6a8-cd1b546d6d51\") " pod="opendatahub/test-curl-7415-140008656553424" Jul 23 18:30:40.458948 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:40.458917 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdbcs\" (UniqueName: \"kubernetes.io/projected/eebcdeb3-6f61-453c-b6a8-cd1b546d6d51-kube-api-access-tdbcs\") pod \"test-curl-7415-140008656553424\" (UID: \"eebcdeb3-6f61-453c-b6a8-cd1b546d6d51\") " pod="opendatahub/test-curl-7415-140008656553424" Jul 23 18:30:40.637577 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:40.637474 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-140008656553424" Jul 23 18:30:40.777471 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:40.777433 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7415-140008656553424"] Jul 23 18:30:40.781164 ip-10-0-137-82 kubenswrapper[2565]: W0723 18:30:40.781120 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeebcdeb3_6f61_453c_b6a8_cd1b546d6d51.slice/crio-7c78ca5de3b00d0299e41fdf62f218efaa92a66858831357af22af6a086abe61 WatchSource:0}: Error finding container 7c78ca5de3b00d0299e41fdf62f218efaa92a66858831357af22af6a086abe61: Status 404 returned error can't find the container with id 7c78ca5de3b00d0299e41fdf62f218efaa92a66858831357af22af6a086abe61 Jul 23 18:30:40.915104 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:40.915013 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7415-140008656553424" event={"ID":"eebcdeb3-6f61-453c-b6a8-cd1b546d6d51","Type":"ContainerStarted","Data":"7c78ca5de3b00d0299e41fdf62f218efaa92a66858831357af22af6a086abe61"} Jul 23 18:30:41.921763 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:41.921727 2565 generic.go:356] "Generic (PLEG): container finished" podID="eebcdeb3-6f61-453c-b6a8-cd1b546d6d51" containerID="0d85cfa088243bf38e0ee6753d54336d05c622e10cbf2fb44aec730607c49641" exitCode=0 Jul 23 18:30:41.922249 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:41.921809 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7415-140008656553424" event={"ID":"eebcdeb3-6f61-453c-b6a8-cd1b546d6d51","Type":"ContainerDied","Data":"0d85cfa088243bf38e0ee6753d54336d05c622e10cbf2fb44aec730607c49641"} Jul 23 18:30:43.070615 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:43.070581 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-140008656553424" Jul 23 18:30:43.091152 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:43.091121 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7415-140008656553424_eebcdeb3-6f61-453c-b6a8-cd1b546d6d51/test-curl-7415-140008656553424/0.log" Jul 23 18:30:43.117394 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:43.117355 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7415-140008656553424"] Jul 23 18:30:43.121840 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:43.121810 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7415-140008656553424"] Jul 23 18:30:43.171835 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:43.171804 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdbcs\" (UniqueName: \"kubernetes.io/projected/eebcdeb3-6f61-453c-b6a8-cd1b546d6d51-kube-api-access-tdbcs\") pod \"eebcdeb3-6f61-453c-b6a8-cd1b546d6d51\" (UID: \"eebcdeb3-6f61-453c-b6a8-cd1b546d6d51\") " Jul 23 18:30:43.173827 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:43.173800 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eebcdeb3-6f61-453c-b6a8-cd1b546d6d51-kube-api-access-tdbcs" (OuterVolumeSpecName: "kube-api-access-tdbcs") pod "eebcdeb3-6f61-453c-b6a8-cd1b546d6d51" (UID: "eebcdeb3-6f61-453c-b6a8-cd1b546d6d51"). InnerVolumeSpecName "kube-api-access-tdbcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:30:43.272739 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:43.272707 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdbcs\" (UniqueName: \"kubernetes.io/projected/eebcdeb3-6f61-453c-b6a8-cd1b546d6d51-kube-api-access-tdbcs\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:30:43.934538 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:43.934509 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c78ca5de3b00d0299e41fdf62f218efaa92a66858831357af22af6a086abe61" Jul 23 18:30:43.934538 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:43.934537 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-140008656553424" Jul 23 18:30:43.945002 ip-10-0-137-82 kubenswrapper[2565]: E0723 18:30:43.944966 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7415-140008656553424\" is forbidden: User \"system:node:ip-10-0-137-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-82.ec2.internal' and this object" podUID="eebcdeb3-6f61-453c-b6a8-cd1b546d6d51" pod="opendatahub/test-curl-7415-140008656553424" Jul 23 18:30:44.405549 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:44.405504 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eebcdeb3-6f61-453c-b6a8-cd1b546d6d51" path="/var/lib/kubelet/pods/eebcdeb3-6f61-453c-b6a8-cd1b546d6d51/volumes" Jul 23 18:30:46.456121 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:46.456082 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4d95fb-s9d8c"] Jul 23 18:30:46.456500 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:46.456484 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eebcdeb3-6f61-453c-b6a8-cd1b546d6d51" containerName="test-curl-7415-140008656553424" Jul 23 18:30:46.456500 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:46.456497 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="eebcdeb3-6f61-453c-b6a8-cd1b546d6d51" containerName="test-curl-7415-140008656553424" Jul 23 18:30:46.456573 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:46.456560 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="eebcdeb3-6f61-453c-b6a8-cd1b546d6d51" containerName="test-curl-7415-140008656553424" Jul 23 18:30:46.459883 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:46.459866 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4d95fb-s9d8c" Jul 23 18:30:46.463168 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:46.463145 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5lmm\"" Jul 23 18:30:46.468837 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:46.468813 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4d95fb-s9d8c"] Jul 23 18:30:46.499371 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:46.499334 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b20e6117-ef8c-4b98-8ba1-024fcb4c1353-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-4d95fb-s9d8c\" (UID: \"b20e6117-ef8c-4b98-8ba1-024fcb4c1353\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4d95fb-s9d8c" Jul 23 18:30:46.600840 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:46.600798 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b20e6117-ef8c-4b98-8ba1-024fcb4c1353-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-4d95fb-s9d8c\" (UID: \"b20e6117-ef8c-4b98-8ba1-024fcb4c1353\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4d95fb-s9d8c" Jul 23 18:30:46.601444 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:46.601412 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b20e6117-ef8c-4b98-8ba1-024fcb4c1353-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-4d95fb-s9d8c\" (UID: \"b20e6117-ef8c-4b98-8ba1-024fcb4c1353\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4d95fb-s9d8c" Jul 23 18:30:46.771567 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:46.771537 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4d95fb-s9d8c" Jul 23 18:30:46.910717 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:46.910693 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4d95fb-s9d8c"] Jul 23 18:30:46.913041 ip-10-0-137-82 kubenswrapper[2565]: W0723 18:30:46.913016 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb20e6117_ef8c_4b98_8ba1_024fcb4c1353.slice/crio-6a719f1152fb3255edf9effd79424eb52d0557ca64905090578b8565fc2494ee WatchSource:0}: Error finding container 6a719f1152fb3255edf9effd79424eb52d0557ca64905090578b8565fc2494ee: Status 404 returned error can't find the container with id 6a719f1152fb3255edf9effd79424eb52d0557ca64905090578b8565fc2494ee Jul 23 18:30:46.950288 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:46.950252 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4d95fb-s9d8c" event={"ID":"b20e6117-ef8c-4b98-8ba1-024fcb4c1353","Type":"ContainerStarted","Data":"6a719f1152fb3255edf9effd79424eb52d0557ca64905090578b8565fc2494ee"} Jul 23 18:30:47.956853 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:47.956762 2565 generic.go:356] "Generic (PLEG): container finished" podID="b20e6117-ef8c-4b98-8ba1-024fcb4c1353" containerID="b2b67226560729e508e044cf6b60bbc271863dc0bc53f88cb134ebb96cc39ebe" exitCode=0 Jul 23 18:30:47.957195 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:47.956850 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4d95fb-s9d8c" event={"ID":"b20e6117-ef8c-4b98-8ba1-024fcb4c1353","Type":"ContainerDied","Data":"b2b67226560729e508e044cf6b60bbc271863dc0bc53f88cb134ebb96cc39ebe"} Jul 23 18:30:49.095162 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:49.095136 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4d95fb-s9d8c" Jul 23 18:30:49.121379 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:49.121355 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b20e6117-ef8c-4b98-8ba1-024fcb4c1353-maas-api-service-ca\") pod \"b20e6117-ef8c-4b98-8ba1-024fcb4c1353\" (UID: \"b20e6117-ef8c-4b98-8ba1-024fcb4c1353\") " Jul 23 18:30:49.121706 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:49.121686 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b20e6117-ef8c-4b98-8ba1-024fcb4c1353-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b20e6117-ef8c-4b98-8ba1-024fcb4c1353" (UID: "b20e6117-ef8c-4b98-8ba1-024fcb4c1353"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:30:49.222725 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:49.222628 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b20e6117-ef8c-4b98-8ba1-024fcb4c1353-maas-api-service-ca\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:30:49.968749 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:49.968714 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4d95fb-s9d8c" event={"ID":"b20e6117-ef8c-4b98-8ba1-024fcb4c1353","Type":"ContainerDied","Data":"6a719f1152fb3255edf9effd79424eb52d0557ca64905090578b8565fc2494ee"} Jul 23 18:30:49.968749 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:49.968737 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4d95fb-s9d8c" Jul 23 18:30:49.968960 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:49.968763 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a719f1152fb3255edf9effd79424eb52d0557ca64905090578b8565fc2494ee" Jul 23 18:30:59.218238 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:59.218187 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp"] Jul 23 18:30:59.218736 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:59.218501 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp" podUID="ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2" containerName="maas-api" containerID="cri-o://ccf826b5a0411f6a37bb61c74c05f18849e60a5fcb1b25ee6d3d384ae86af16a" gracePeriod=30 Jul 23 18:30:59.249934 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:59.249907 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4d95fb-29747190-tz4x9"] Jul 23 18:30:59.253150 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:59.253128 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4d95fb-29747190-tz4x9"] Jul 23 18:30:59.478807 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:59.478749 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp" Jul 23 18:30:59.615293 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:59.615257 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qngm7\" (UniqueName: \"kubernetes.io/projected/ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2-kube-api-access-qngm7\") pod \"ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2\" (UID: \"ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2\") " Jul 23 18:30:59.615466 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:59.615350 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2-openshift-service-ca\") pod \"ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2\" (UID: \"ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2\") " Jul 23 18:30:59.615466 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:59.615402 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2-maas-api-tls\") pod \"ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2\" (UID: \"ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2\") " Jul 23 18:30:59.615761 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:59.615730 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2" (UID: "ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:30:59.617463 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:59.617442 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2" (UID: "ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:30:59.617558 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:59.617500 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2-kube-api-access-qngm7" (OuterVolumeSpecName: "kube-api-access-qngm7") pod "ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2" (UID: "ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2"). InnerVolumeSpecName "kube-api-access-qngm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:30:59.717117 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:59.717082 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2-openshift-service-ca\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:30:59.717117 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:59.717115 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2-maas-api-tls\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:30:59.717117 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:30:59.717125 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qngm7\" (UniqueName: \"kubernetes.io/projected/ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2-kube-api-access-qngm7\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:31:00.020777 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:00.020737 2565 generic.go:356] "Generic (PLEG): container finished" podID="ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2" containerID="ccf826b5a0411f6a37bb61c74c05f18849e60a5fcb1b25ee6d3d384ae86af16a" exitCode=0 Jul 23 18:31:00.020951 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:00.020784 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp" Jul 23 18:31:00.020951 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:00.020801 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp" event={"ID":"ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2","Type":"ContainerDied","Data":"ccf826b5a0411f6a37bb61c74c05f18849e60a5fcb1b25ee6d3d384ae86af16a"} Jul 23 18:31:00.020951 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:00.020872 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp" event={"ID":"ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2","Type":"ContainerDied","Data":"d3f950fdd1096714715842bc805b46e05be600273dab4180e52456a1c2e4d246"} Jul 23 18:31:00.020951 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:00.020903 2565 scope.go:117] "RemoveContainer" containerID="ccf826b5a0411f6a37bb61c74c05f18849e60a5fcb1b25ee6d3d384ae86af16a" Jul 23 18:31:00.032793 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:00.032772 2565 scope.go:117] "RemoveContainer" containerID="ccf826b5a0411f6a37bb61c74c05f18849e60a5fcb1b25ee6d3d384ae86af16a" Jul 23 18:31:00.033107 ip-10-0-137-82 kubenswrapper[2565]: E0723 18:31:00.033084 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccf826b5a0411f6a37bb61c74c05f18849e60a5fcb1b25ee6d3d384ae86af16a\": container with ID starting with ccf826b5a0411f6a37bb61c74c05f18849e60a5fcb1b25ee6d3d384ae86af16a not found: ID does not exist" containerID="ccf826b5a0411f6a37bb61c74c05f18849e60a5fcb1b25ee6d3d384ae86af16a" Jul 23 18:31:00.033180 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:00.033121 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccf826b5a0411f6a37bb61c74c05f18849e60a5fcb1b25ee6d3d384ae86af16a"} err="failed to get container status \"ccf826b5a0411f6a37bb61c74c05f18849e60a5fcb1b25ee6d3d384ae86af16a\": rpc error: code = NotFound desc = could not find container \"ccf826b5a0411f6a37bb61c74c05f18849e60a5fcb1b25ee6d3d384ae86af16a\": container with ID starting with ccf826b5a0411f6a37bb61c74c05f18849e60a5fcb1b25ee6d3d384ae86af16a not found: ID does not exist" Jul 23 18:31:00.043748 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:00.043727 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp"] Jul 23 18:31:00.048199 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:00.048170 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4d95fb-765fbfb796-dhlkp"] Jul 23 18:31:00.407081 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:00.407002 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2" path="/var/lib/kubelet/pods/ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2/volumes" Jul 23 18:31:00.407421 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:00.407290 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8d45bea-1e73-4c4a-87d9-b23711bd80dd" path="/var/lib/kubelet/pods/d8d45bea-1e73-4c4a-87d9-b23711bd80dd/volumes" Jul 23 18:31:01.570302 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:01.570265 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4d95fb-s9d8c"] Jul 23 18:31:01.573749 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:01.573725 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4d95fb-s9d8c"] Jul 23 18:31:02.405969 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:02.405926 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b20e6117-ef8c-4b98-8ba1-024fcb4c1353" path="/var/lib/kubelet/pods/b20e6117-ef8c-4b98-8ba1-024fcb4c1353/volumes" Jul 23 18:31:09.035265 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:09.035229 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e1d86b-rqc22"] Jul 23 18:31:09.035667 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:09.035609 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2" containerName="maas-api" Jul 23 18:31:09.035667 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:09.035622 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2" containerName="maas-api" Jul 23 18:31:09.035667 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:09.035641 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b20e6117-ef8c-4b98-8ba1-024fcb4c1353" containerName="revoke-keys" Jul 23 18:31:09.035667 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:09.035647 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b20e6117-ef8c-4b98-8ba1-024fcb4c1353" containerName="revoke-keys" Jul 23 18:31:09.035799 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:09.035720 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae32480f-8ce0-4ace-a3f8-38ccb2d9ddb2" containerName="maas-api" Jul 23 18:31:09.035799 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:09.035730 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b20e6117-ef8c-4b98-8ba1-024fcb4c1353" containerName="revoke-keys" Jul 23 18:31:09.040545 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:09.040526 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e1d86b-rqc22" Jul 23 18:31:09.043174 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:09.043146 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5lmm\"" Jul 23 18:31:09.046655 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:09.046626 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e1d86b-rqc22"] Jul 23 18:31:09.208679 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:09.208640 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa1b6cf3-e252-4365-8102-9820fea46133-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-e1d86b-rqc22\" (UID: \"aa1b6cf3-e252-4365-8102-9820fea46133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e1d86b-rqc22" Jul 23 18:31:09.309313 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:09.309216 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa1b6cf3-e252-4365-8102-9820fea46133-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-e1d86b-rqc22\" (UID: \"aa1b6cf3-e252-4365-8102-9820fea46133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e1d86b-rqc22" Jul 23 18:31:09.309964 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:09.309939 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa1b6cf3-e252-4365-8102-9820fea46133-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-e1d86b-rqc22\" (UID: \"aa1b6cf3-e252-4365-8102-9820fea46133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e1d86b-rqc22" Jul 23 18:31:09.352438 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:09.352405 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e1d86b-rqc22" Jul 23 18:31:09.491083 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:09.491020 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e1d86b-rqc22"] Jul 23 18:31:09.493467 ip-10-0-137-82 kubenswrapper[2565]: W0723 18:31:09.493442 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa1b6cf3_e252_4365_8102_9820fea46133.slice/crio-52631d006743588b5608d9ed814d5ca0739dfa036f43f9dfc901081b9bec5091 WatchSource:0}: Error finding container 52631d006743588b5608d9ed814d5ca0739dfa036f43f9dfc901081b9bec5091: Status 404 returned error can't find the container with id 52631d006743588b5608d9ed814d5ca0739dfa036f43f9dfc901081b9bec5091 Jul 23 18:31:10.071921 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:10.071887 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e1d86b-rqc22" event={"ID":"aa1b6cf3-e252-4365-8102-9820fea46133","Type":"ContainerStarted","Data":"9e9ab69affdf13611ad92f8390a845f8d4c25494e9db43bc5cb9e22bc7e2e72f"} Jul 23 18:31:10.072313 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:10.071943 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e1d86b-rqc22" event={"ID":"aa1b6cf3-e252-4365-8102-9820fea46133","Type":"ContainerStarted","Data":"52631d006743588b5608d9ed814d5ca0739dfa036f43f9dfc901081b9bec5091"} Jul 23 18:31:10.092245 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:10.092189 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e1d86b-rqc22" podStartSLOduration=1.092174685 podStartE2EDuration="1.092174685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:31:10.090667799 +0000 UTC m=+2814.168971152" watchObservedRunningTime="2026-07-23 18:31:10.092174685 +0000 UTC m=+2814.170478037" Jul 23 18:31:11.078480 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:11.078450 2565 generic.go:356] "Generic (PLEG): container finished" podID="aa1b6cf3-e252-4365-8102-9820fea46133" containerID="9e9ab69affdf13611ad92f8390a845f8d4c25494e9db43bc5cb9e22bc7e2e72f" exitCode=0 Jul 23 18:31:11.078914 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:11.078540 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e1d86b-rqc22" event={"ID":"aa1b6cf3-e252-4365-8102-9820fea46133","Type":"ContainerDied","Data":"9e9ab69affdf13611ad92f8390a845f8d4c25494e9db43bc5cb9e22bc7e2e72f"} Jul 23 18:31:12.220264 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:12.220232 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e1d86b-rqc22" Jul 23 18:31:12.338267 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:12.338155 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa1b6cf3-e252-4365-8102-9820fea46133-maas-api-service-ca\") pod \"aa1b6cf3-e252-4365-8102-9820fea46133\" (UID: \"aa1b6cf3-e252-4365-8102-9820fea46133\") " Jul 23 18:31:12.338548 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:12.338523 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa1b6cf3-e252-4365-8102-9820fea46133-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aa1b6cf3-e252-4365-8102-9820fea46133" (UID: "aa1b6cf3-e252-4365-8102-9820fea46133"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:31:12.439152 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:12.439119 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa1b6cf3-e252-4365-8102-9820fea46133-maas-api-service-ca\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:31:13.089416 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:13.089382 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e1d86b-rqc22" Jul 23 18:31:13.089616 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:13.089386 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e1d86b-rqc22" event={"ID":"aa1b6cf3-e252-4365-8102-9820fea46133","Type":"ContainerDied","Data":"52631d006743588b5608d9ed814d5ca0739dfa036f43f9dfc901081b9bec5091"} Jul 23 18:31:13.089616 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:13.089525 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52631d006743588b5608d9ed814d5ca0739dfa036f43f9dfc901081b9bec5091" Jul 23 18:31:19.402330 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:19.402288 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw"] Jul 23 18:31:19.402833 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:19.402537 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw" podUID="0ce25c79-d0a8-46f1-9008-d236136d56f0" containerName="maas-api" containerID="cri-o://e9845637a476ec6867847f9ecbbc0f8e72fb8a83ce12ec731c0ebae31e918d6b" gracePeriod=30 Jul 23 18:31:19.442956 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:19.442927 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e1d86b-29747190-hscnb"] Jul 23 18:31:19.444512 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:19.444487 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e1d86b-29747190-hscnb"] Jul 23 18:31:19.663252 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:19.663196 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw" Jul 23 18:31:19.706980 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:19.706950 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnc62\" (UniqueName: \"kubernetes.io/projected/0ce25c79-d0a8-46f1-9008-d236136d56f0-kube-api-access-fnc62\") pod \"0ce25c79-d0a8-46f1-9008-d236136d56f0\" (UID: \"0ce25c79-d0a8-46f1-9008-d236136d56f0\") " Jul 23 18:31:19.707169 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:19.707030 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce25c79-d0a8-46f1-9008-d236136d56f0-openshift-service-ca\") pod \"0ce25c79-d0a8-46f1-9008-d236136d56f0\" (UID: \"0ce25c79-d0a8-46f1-9008-d236136d56f0\") " Jul 23 18:31:19.707169 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:19.707075 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ce25c79-d0a8-46f1-9008-d236136d56f0-maas-api-tls\") pod \"0ce25c79-d0a8-46f1-9008-d236136d56f0\" (UID: \"0ce25c79-d0a8-46f1-9008-d236136d56f0\") " Jul 23 18:31:19.707494 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:19.707464 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ce25c79-d0a8-46f1-9008-d236136d56f0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0ce25c79-d0a8-46f1-9008-d236136d56f0" (UID: "0ce25c79-d0a8-46f1-9008-d236136d56f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:31:19.709145 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:19.709113 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ce25c79-d0a8-46f1-9008-d236136d56f0-kube-api-access-fnc62" (OuterVolumeSpecName: "kube-api-access-fnc62") pod "0ce25c79-d0a8-46f1-9008-d236136d56f0" (UID: "0ce25c79-d0a8-46f1-9008-d236136d56f0"). InnerVolumeSpecName "kube-api-access-fnc62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:31:19.709145 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:19.709124 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ce25c79-d0a8-46f1-9008-d236136d56f0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0ce25c79-d0a8-46f1-9008-d236136d56f0" (UID: "0ce25c79-d0a8-46f1-9008-d236136d56f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:31:19.807763 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:19.807723 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnc62\" (UniqueName: \"kubernetes.io/projected/0ce25c79-d0a8-46f1-9008-d236136d56f0-kube-api-access-fnc62\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:31:19.807763 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:19.807756 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce25c79-d0a8-46f1-9008-d236136d56f0-openshift-service-ca\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:31:19.807763 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:19.807772 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ce25c79-d0a8-46f1-9008-d236136d56f0-maas-api-tls\") on node \"ip-10-0-137-82.ec2.internal\" DevicePath \"\"" Jul 23 18:31:20.128747 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:20.128707 2565 generic.go:356] "Generic (PLEG): container finished" podID="0ce25c79-d0a8-46f1-9008-d236136d56f0" containerID="e9845637a476ec6867847f9ecbbc0f8e72fb8a83ce12ec731c0ebae31e918d6b" exitCode=0 Jul 23 18:31:20.128945 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:20.128761 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw" Jul 23 18:31:20.128945 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:20.128792 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw" event={"ID":"0ce25c79-d0a8-46f1-9008-d236136d56f0","Type":"ContainerDied","Data":"e9845637a476ec6867847f9ecbbc0f8e72fb8a83ce12ec731c0ebae31e918d6b"} Jul 23 18:31:20.128945 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:20.128866 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw" event={"ID":"0ce25c79-d0a8-46f1-9008-d236136d56f0","Type":"ContainerDied","Data":"7797dc89e49f0b751606d6e488867f2f78bf1de71cd5a51d110a337c122273b5"} Jul 23 18:31:20.128945 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:20.128901 2565 scope.go:117] "RemoveContainer" containerID="e9845637a476ec6867847f9ecbbc0f8e72fb8a83ce12ec731c0ebae31e918d6b" Jul 23 18:31:20.137833 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:20.137816 2565 scope.go:117] "RemoveContainer" containerID="e9845637a476ec6867847f9ecbbc0f8e72fb8a83ce12ec731c0ebae31e918d6b" Jul 23 18:31:20.138063 ip-10-0-137-82 kubenswrapper[2565]: E0723 18:31:20.138040 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9845637a476ec6867847f9ecbbc0f8e72fb8a83ce12ec731c0ebae31e918d6b\": container with ID starting with e9845637a476ec6867847f9ecbbc0f8e72fb8a83ce12ec731c0ebae31e918d6b not found: ID does not exist" containerID="e9845637a476ec6867847f9ecbbc0f8e72fb8a83ce12ec731c0ebae31e918d6b" Jul 23 18:31:20.138130 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:20.138078 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9845637a476ec6867847f9ecbbc0f8e72fb8a83ce12ec731c0ebae31e918d6b"} err="failed to get container status \"e9845637a476ec6867847f9ecbbc0f8e72fb8a83ce12ec731c0ebae31e918d6b\": rpc error: code = NotFound desc = could not find container \"e9845637a476ec6867847f9ecbbc0f8e72fb8a83ce12ec731c0ebae31e918d6b\": container with ID starting with e9845637a476ec6867847f9ecbbc0f8e72fb8a83ce12ec731c0ebae31e918d6b not found: ID does not exist" Jul 23 18:31:20.149770 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:20.149744 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw"] Jul 23 18:31:20.153186 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:20.153165 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e1d86b-6474c8f67f-mdkhw"] Jul 23 18:31:20.406197 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:20.406120 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ce25c79-d0a8-46f1-9008-d236136d56f0" path="/var/lib/kubelet/pods/0ce25c79-d0a8-46f1-9008-d236136d56f0/volumes" Jul 23 18:31:20.406533 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:20.406383 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3998af9-9811-4f51-84dd-2ef45e51d626" path="/var/lib/kubelet/pods/c3998af9-9811-4f51-84dd-2ef45e51d626/volumes" Jul 23 18:31:24.127449 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:24.127410 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e1d86b-rqc22"] Jul 23 18:31:24.129562 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:24.129536 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e1d86b-rqc22"] Jul 23 18:31:24.406024 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:24.405941 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa1b6cf3-e252-4365-8102-9820fea46133" path="/var/lib/kubelet/pods/aa1b6cf3-e252-4365-8102-9820fea46133/volumes" Jul 23 18:31:37.297793 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:37.297761 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-sp7bq_fa68142b-8dd3-4e3f-aa2f-23a6598568e1/manager/0.log" Jul 23 18:31:37.416523 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:37.416489 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-6c4zk_d0c8c911-4f29-4cc0-b055-514528b1afd3/manager/0.log" Jul 23 18:31:38.149516 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:38.149485 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-547c58d5d9-pfr5x_740d1253-7e70-4b58-9c93-10cbd14d29d1/copy-manifests/0.log" Jul 23 18:31:38.157430 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:38.157358 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-547c58d5d9-pfr5x_740d1253-7e70-4b58-9c93-10cbd14d29d1/manager/0.log" Jul 23 18:31:38.521205 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:38.521175 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6755c5b8f8-xtm6x_ca4b9476-9cf2-44f1-95b3-51215376a655/maas-api/0.log" Jul 23 18:31:38.743177 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:38.743144 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29747175-gr796_a580b0b0-f010-4c4c-aad2-43590ad632ac/cleanup/0.log" Jul 23 18:31:38.851327 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:38.851239 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29747190-kc7kw_c8bab927-792a-4fbe-8d53-522524b92670/cleanup/0.log" Jul 23 18:31:40.845553 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:40.845519 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-9874m_cf6df2af-6e72-4bd7-aa51-04dcd0fc1bbe/manager/0.log" Jul 23 18:31:42.162430 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:42.162400 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-kq59n_3d5a8e69-cc15-4912-952b-272908244643/payload-processing/0.log" Jul 23 18:31:42.287969 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:42.287933 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7b4c7f49cd-9kl9k_13bc8906-58cc-4237-88d6-beade4de9408/router/0.log" Jul 23 18:31:42.996941 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:42.996909 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9jb9q_64bd615c-b608-423d-a4cd-d6122be3c236/main/0.log" Jul 23 18:31:43.112381 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:43.112334 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-zm89g_d7bb31f1-07ea-4418-b744-fb09ba8f3f53/main/0.log" Jul 23 18:31:51.628064 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:51.628032 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2j6vd_39d55095-246a-4ecf-bb04-36a9a8c92da4/global-pull-secret-syncer/0.log" Jul 23 18:31:51.929315 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:51.929237 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kcd4h_4a324aac-61f5-46a9-b847-15730ba25434/konnectivity-agent/0.log" Jul 23 18:31:52.023491 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:52.023463 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-82.ec2.internal_175ceae0d598475e91b574d725c1c32d/haproxy/0.log" Jul 23 18:31:57.247746 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:57.247711 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-9874m_cf6df2af-6e72-4bd7-aa51-04dcd0fc1bbe/manager/0.log" Jul 23 18:31:59.084084 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:59.084051 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-rwqpw_e29fa838-5b80-4246-90f9-a9095e507279/cluster-monitoring-operator/0.log" Jul 23 18:31:59.390160 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:59.390069 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hcvz7_1eb42749-defc-4272-a671-4569a9705519/node-exporter/0.log" Jul 23 18:31:59.411968 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:59.411942 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hcvz7_1eb42749-defc-4272-a671-4569a9705519/kube-rbac-proxy/0.log" Jul 23 18:31:59.432438 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:59.432415 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hcvz7_1eb42749-defc-4272-a671-4569a9705519/init-textfile/0.log" Jul 23 18:31:59.863720 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:31:59.863689 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-2tmcd_f608c6e6-9cbe-45f5-b2d6-b1f19b846bbd/prometheus-operator-admission-webhook/0.log" Jul 23 18:32:01.350561 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:01.350529 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-2mbb6_8b5828c5-b41b-4b6b-8300-7879fd076ff9/networking-console-plugin/0.log" Jul 23 18:32:02.010711 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.010677 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-m7k4j_1cca3236-5c90-4abb-9b9b-39e40b1903c3/console-operator/1.log" Jul 23 18:32:02.019633 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.019591 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-m7k4j_1cca3236-5c90-4abb-9b9b-39e40b1903c3/console-operator/2.log" Jul 23 18:32:02.194152 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.194112 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-v5cbl/perf-node-gather-daemonset-r5rfx"] Jul 23 18:32:02.194502 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.194487 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa1b6cf3-e252-4365-8102-9820fea46133" containerName="revoke-keys" Jul 23 18:32:02.194546 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.194503 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa1b6cf3-e252-4365-8102-9820fea46133" containerName="revoke-keys" Jul 23 18:32:02.194546 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.194527 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ce25c79-d0a8-46f1-9008-d236136d56f0" containerName="maas-api" Jul 23 18:32:02.194546 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.194532 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ce25c79-d0a8-46f1-9008-d236136d56f0" containerName="maas-api" Jul 23 18:32:02.194669 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.194610 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ce25c79-d0a8-46f1-9008-d236136d56f0" containerName="maas-api" Jul 23 18:32:02.194669 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.194621 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa1b6cf3-e252-4365-8102-9820fea46133" containerName="revoke-keys" Jul 23 18:32:02.197702 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.197687 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v5cbl/perf-node-gather-daemonset-r5rfx" Jul 23 18:32:02.207391 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.207373 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v5cbl\"/\"openshift-service-ca.crt\"" Jul 23 18:32:02.207748 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.207733 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v5cbl\"/\"kube-root-ca.crt\"" Jul 23 18:32:02.208456 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.208441 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-v5cbl\"/\"default-dockercfg-qc65b\"" Jul 23 18:32:02.216107 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.216083 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v5cbl/perf-node-gather-daemonset-r5rfx"] Jul 23 18:32:02.286503 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.286471 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3bc98347-02bc-4a62-87f6-e3c94b13c5f2-proc\") pod \"perf-node-gather-daemonset-r5rfx\" (UID: \"3bc98347-02bc-4a62-87f6-e3c94b13c5f2\") " pod="openshift-must-gather-v5cbl/perf-node-gather-daemonset-r5rfx" Jul 23 18:32:02.286724 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.286536 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3bc98347-02bc-4a62-87f6-e3c94b13c5f2-podres\") pod \"perf-node-gather-daemonset-r5rfx\" (UID: \"3bc98347-02bc-4a62-87f6-e3c94b13c5f2\") " pod="openshift-must-gather-v5cbl/perf-node-gather-daemonset-r5rfx" Jul 23 18:32:02.286724 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.286562 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrmxs\" (UniqueName: \"kubernetes.io/projected/3bc98347-02bc-4a62-87f6-e3c94b13c5f2-kube-api-access-lrmxs\") pod \"perf-node-gather-daemonset-r5rfx\" (UID: \"3bc98347-02bc-4a62-87f6-e3c94b13c5f2\") " pod="openshift-must-gather-v5cbl/perf-node-gather-daemonset-r5rfx" Jul 23 18:32:02.286724 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.286591 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3bc98347-02bc-4a62-87f6-e3c94b13c5f2-lib-modules\") pod \"perf-node-gather-daemonset-r5rfx\" (UID: \"3bc98347-02bc-4a62-87f6-e3c94b13c5f2\") " pod="openshift-must-gather-v5cbl/perf-node-gather-daemonset-r5rfx" Jul 23 18:32:02.286724 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.286691 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bc98347-02bc-4a62-87f6-e3c94b13c5f2-sys\") pod \"perf-node-gather-daemonset-r5rfx\" (UID: \"3bc98347-02bc-4a62-87f6-e3c94b13c5f2\") " pod="openshift-must-gather-v5cbl/perf-node-gather-daemonset-r5rfx" Jul 23 18:32:02.387337 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.387301 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3bc98347-02bc-4a62-87f6-e3c94b13c5f2-podres\") pod \"perf-node-gather-daemonset-r5rfx\" (UID: \"3bc98347-02bc-4a62-87f6-e3c94b13c5f2\") " pod="openshift-must-gather-v5cbl/perf-node-gather-daemonset-r5rfx" Jul 23 18:32:02.387801 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.387344 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrmxs\" (UniqueName: \"kubernetes.io/projected/3bc98347-02bc-4a62-87f6-e3c94b13c5f2-kube-api-access-lrmxs\") pod \"perf-node-gather-daemonset-r5rfx\" (UID: \"3bc98347-02bc-4a62-87f6-e3c94b13c5f2\") " pod="openshift-must-gather-v5cbl/perf-node-gather-daemonset-r5rfx" Jul 23 18:32:02.387801 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.387372 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3bc98347-02bc-4a62-87f6-e3c94b13c5f2-lib-modules\") pod \"perf-node-gather-daemonset-r5rfx\" (UID: \"3bc98347-02bc-4a62-87f6-e3c94b13c5f2\") " pod="openshift-must-gather-v5cbl/perf-node-gather-daemonset-r5rfx" Jul 23 18:32:02.387801 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.387415 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bc98347-02bc-4a62-87f6-e3c94b13c5f2-sys\") pod \"perf-node-gather-daemonset-r5rfx\" (UID: \"3bc98347-02bc-4a62-87f6-e3c94b13c5f2\") " pod="openshift-must-gather-v5cbl/perf-node-gather-daemonset-r5rfx" Jul 23 18:32:02.387801 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.387436 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3bc98347-02bc-4a62-87f6-e3c94b13c5f2-proc\") pod \"perf-node-gather-daemonset-r5rfx\" (UID: \"3bc98347-02bc-4a62-87f6-e3c94b13c5f2\") " pod="openshift-must-gather-v5cbl/perf-node-gather-daemonset-r5rfx" Jul 23 18:32:02.387801 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.387474 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3bc98347-02bc-4a62-87f6-e3c94b13c5f2-podres\") pod \"perf-node-gather-daemonset-r5rfx\" (UID: \"3bc98347-02bc-4a62-87f6-e3c94b13c5f2\") " pod="openshift-must-gather-v5cbl/perf-node-gather-daemonset-r5rfx" Jul 23 18:32:02.387801 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.387503 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bc98347-02bc-4a62-87f6-e3c94b13c5f2-sys\") pod \"perf-node-gather-daemonset-r5rfx\" (UID: \"3bc98347-02bc-4a62-87f6-e3c94b13c5f2\") " pod="openshift-must-gather-v5cbl/perf-node-gather-daemonset-r5rfx" Jul 23 18:32:02.387801 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.387518 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3bc98347-02bc-4a62-87f6-e3c94b13c5f2-proc\") pod \"perf-node-gather-daemonset-r5rfx\" (UID: \"3bc98347-02bc-4a62-87f6-e3c94b13c5f2\") " pod="openshift-must-gather-v5cbl/perf-node-gather-daemonset-r5rfx" Jul 23 18:32:02.387801 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.387568 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3bc98347-02bc-4a62-87f6-e3c94b13c5f2-lib-modules\") pod \"perf-node-gather-daemonset-r5rfx\" (UID: \"3bc98347-02bc-4a62-87f6-e3c94b13c5f2\") " pod="openshift-must-gather-v5cbl/perf-node-gather-daemonset-r5rfx" Jul 23 18:32:02.396550 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.396515 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrmxs\" (UniqueName: \"kubernetes.io/projected/3bc98347-02bc-4a62-87f6-e3c94b13c5f2-kube-api-access-lrmxs\") pod \"perf-node-gather-daemonset-r5rfx\" (UID: \"3bc98347-02bc-4a62-87f6-e3c94b13c5f2\") " pod="openshift-must-gather-v5cbl/perf-node-gather-daemonset-r5rfx" Jul 23 18:32:02.507626 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.507569 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v5cbl/perf-node-gather-daemonset-r5rfx" Jul 23 18:32:02.637927 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.637900 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v5cbl/perf-node-gather-daemonset-r5rfx"] Jul 23 18:32:02.639793 ip-10-0-137-82 kubenswrapper[2565]: W0723 18:32:02.639767 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3bc98347_02bc_4a62_87f6_e3c94b13c5f2.slice/crio-bf837d08e8d06f16dcd3d6e00d2375ffa1c8e5f1d3c5ecbad94c36e1ec1db57f WatchSource:0}: Error finding container bf837d08e8d06f16dcd3d6e00d2375ffa1c8e5f1d3c5ecbad94c36e1ec1db57f: Status 404 returned error can't find the container with id bf837d08e8d06f16dcd3d6e00d2375ffa1c8e5f1d3c5ecbad94c36e1ec1db57f Jul 23 18:32:02.683492 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.683466 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6b4ff9cd5d-9dgmq_a2f67767-4863-4a8a-9999-99875fd52c19/console/0.log" Jul 23 18:32:02.723869 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:02.723849 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-jrsv7_6a7d3768-fb48-410a-b5cb-e936bdfb9baf/download-server/0.log" Jul 23 18:32:03.357498 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:03.357459 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v5cbl/perf-node-gather-daemonset-r5rfx" event={"ID":"3bc98347-02bc-4a62-87f6-e3c94b13c5f2","Type":"ContainerStarted","Data":"682fec5fe4c6265c74772a58b4121809ca271041d72e359b6bc2bb223cac80bc"} Jul 23 18:32:03.357721 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:03.357618 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-v5cbl/perf-node-gather-daemonset-r5rfx" Jul 23 18:32:03.357721 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:03.357647 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v5cbl/perf-node-gather-daemonset-r5rfx" event={"ID":"3bc98347-02bc-4a62-87f6-e3c94b13c5f2","Type":"ContainerStarted","Data":"bf837d08e8d06f16dcd3d6e00d2375ffa1c8e5f1d3c5ecbad94c36e1ec1db57f"} Jul 23 18:32:03.361678 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:03.361654 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-dqmhz_211a908c-6578-4515-a867-8da6b0125dea/volume-data-source-validator/0.log" Jul 23 18:32:03.375507 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:03.375462 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-v5cbl/perf-node-gather-daemonset-r5rfx" podStartSLOduration=1.37544823 podStartE2EDuration="1.37544823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:32:03.37291145 +0000 UTC m=+2867.451214802" watchObservedRunningTime="2026-07-23 18:32:03.37544823 +0000 UTC m=+2867.453751583" Jul 23 18:32:04.493817 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:04.493790 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vsw5b_35c70c4c-f5f0-49a6-a61f-f8676deeb25a/dns/0.log" Jul 23 18:32:04.513369 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:04.513340 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vsw5b_35c70c4c-f5f0-49a6-a61f-f8676deeb25a/kube-rbac-proxy/0.log" Jul 23 18:32:04.535186 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:04.535163 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6nsmg_69063b0f-0422-463f-833c-5aecf8d331a7/dns-node-resolver/0.log" Jul 23 18:32:05.159797 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:05.159756 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m8496_b11ec86f-5925-4385-aeca-5b5e88f693fa/node-ca/0.log" Jul 23 18:32:06.590723 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:06.590683 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-kq59n_3d5a8e69-cc15-4912-952b-272908244643/payload-processing/0.log" Jul 23 18:32:06.612221 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:06.612183 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7b4c7f49cd-9kl9k_13bc8906-58cc-4237-88d6-beade4de9408/router/0.log" Jul 23 18:32:07.325305 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:07.325274 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sp8s4_5d532709-e70c-4eaf-b4e9-e650237862ea/serve-healthcheck-canary/0.log" Jul 23 18:32:07.815968 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:07.815936 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-n55cw_f590b7bd-8791-41a3-a520-9ba3559d3371/insights-operator/0.log" Jul 23 18:32:07.817301 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:07.817274 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-n55cw_f590b7bd-8791-41a3-a520-9ba3559d3371/insights-operator/1.log" Jul 23 18:32:07.837936 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:07.837913 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9lncb_b73976ea-f0c0-4385-a188-01e56bac9282/kube-rbac-proxy/0.log" Jul 23 18:32:07.859959 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:07.859940 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9lncb_b73976ea-f0c0-4385-a188-01e56bac9282/exporter/0.log" Jul 23 18:32:07.879765 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:07.879740 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9lncb_b73976ea-f0c0-4385-a188-01e56bac9282/extractor/0.log" Jul 23 18:32:09.372737 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:09.372669 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-v5cbl/perf-node-gather-daemonset-r5rfx" Jul 23 18:32:10.154957 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:10.154921 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-sp7bq_fa68142b-8dd3-4e3f-aa2f-23a6598568e1/manager/0.log" Jul 23 18:32:10.504701 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:10.504584 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-6c4zk_d0c8c911-4f29-4cc0-b055-514528b1afd3/manager/0.log" Jul 23 18:32:11.019085 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:11.019054 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-547c58d5d9-pfr5x_740d1253-7e70-4b58-9c93-10cbd14d29d1/manager/0.log" Jul 23 18:32:11.042393 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:11.042366 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-547c58d5d9-pfr5x_740d1253-7e70-4b58-9c93-10cbd14d29d1/copy-manifests/0.log" Jul 23 18:32:12.461566 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:12.461537 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-754bfdc447-jwg9h_5691d164-7cef-4621-a094-4f0879d05407/manager/0.log" Jul 23 18:32:13.465800 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:13.465771 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-kq59n_3d5a8e69-cc15-4912-952b-272908244643/payload-processing/0.log" Jul 23 18:32:13.482376 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:13.482352 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7b4c7f49cd-9kl9k_13bc8906-58cc-4237-88d6-beade4de9408/router/0.log" Jul 23 18:32:15.025265 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:15.025205 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-kq59n_3d5a8e69-cc15-4912-952b-272908244643/payload-processing/0.log" Jul 23 18:32:15.041873 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:15.041841 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7b4c7f49cd-9kl9k_13bc8906-58cc-4237-88d6-beade4de9408/router/0.log" Jul 23 18:32:17.693539 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:17.693500 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-2wzfz_41a5c93d-25d0-4b6b-8978-1e9bf33daa64/migrator/0.log" Jul 23 18:32:17.712946 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:17.712918 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-2wzfz_41a5c93d-25d0-4b6b-8978-1e9bf33daa64/graceful-termination/0.log" Jul 23 18:32:18.165956 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:18.165919 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-xh686_e372e1b3-c8b6-4995-a770-6a970cd7f11e/kube-storage-version-migrator-operator/1.log" Jul 23 18:32:18.167745 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:18.167720 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-xh686_e372e1b3-c8b6-4995-a770-6a970cd7f11e/kube-storage-version-migrator-operator/0.log" Jul 23 18:32:19.589054 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:19.589021 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vcfvv_5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce/kube-multus-additional-cni-plugins/0.log" Jul 23 18:32:19.608715 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:19.608688 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vcfvv_5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce/egress-router-binary-copy/0.log" Jul 23 18:32:19.628852 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:19.628804 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vcfvv_5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce/cni-plugins/0.log" Jul 23 18:32:19.652740 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:19.652718 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vcfvv_5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce/bond-cni-plugin/0.log" Jul 23 18:32:19.675974 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:19.675948 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vcfvv_5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce/routeoverride-cni/0.log" Jul 23 18:32:19.705976 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:19.705949 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vcfvv_5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce/whereabouts-cni-bincopy/0.log" Jul 23 18:32:19.728611 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:19.728583 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vcfvv_5fdc46dc-9f5e-4f6b-817b-fb7993f8e2ce/whereabouts-cni/0.log" Jul 23 18:32:19.779468 ip-10-0-137-82 kubenswrapper[2565]: I0723 18:32:19.779435 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-ggqsr_91f8ae79-5ace-4707-82f0-c5a5770eba19/kube-multus/0.log"