Jul 29 04:01:42.210062 ip-10-0-139-111 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 29 04:01:42.210074 ip-10-0-139-111 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 29 04:01:42.210084 ip-10-0-139-111 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 29 04:01:42.210329 ip-10-0-139-111 systemd[1]: Failed to start Kubernetes Kubelet. Jul 29 04:01:52.634890 ip-10-0-139-111 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Jul 29 04:01:52.634959 ip-10-0-139-111 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot cb2ee790f3d849b5b0a0928d46c8af11 -- Jul 29 04:03:39.042525 ip-10-0-139-111 systemd[1]: Starting Kubernetes Kubelet... Jul 29 04:03:39.367983 ip-10-0-139-111 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 04:03:39.367983 ip-10-0-139-111 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 29 04:03:39.367983 ip-10-0-139-111 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 04:03:39.367983 ip-10-0-139-111 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 29 04:03:39.367983 ip-10-0-139-111 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 04:03:39.369554 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.369474 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 29 04:03:39.375300 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375273 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 04:03:39.375300 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375293 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 04:03:39.375300 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375298 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 04:03:39.375300 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375301 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 04:03:39.375300 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375304 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 04:03:39.375300 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375307 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 04:03:39.375300 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375310 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 04:03:39.375625 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375312 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 04:03:39.375625 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375315 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 04:03:39.375625 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375318 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 04:03:39.375625 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375321 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 04:03:39.375625 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375323 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 04:03:39.375625 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375326 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 04:03:39.375625 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375328 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 04:03:39.375625 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375331 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 04:03:39.375625 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375335 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 04:03:39.375625 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375337 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 04:03:39.375625 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375340 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 04:03:39.375625 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375342 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 04:03:39.375625 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375348 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 04:03:39.375625 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375351 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 04:03:39.375625 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375354 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 04:03:39.375625 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375356 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 04:03:39.375625 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375359 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 04:03:39.375625 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375361 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 04:03:39.375625 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375364 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 04:03:39.376236 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375366 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 04:03:39.376236 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375369 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 04:03:39.376236 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375372 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 04:03:39.376236 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375374 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 04:03:39.376236 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375377 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 04:03:39.376236 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375379 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 04:03:39.376236 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375381 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 04:03:39.376236 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375384 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 04:03:39.376236 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375386 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 04:03:39.376236 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375389 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 04:03:39.376236 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375392 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 04:03:39.376236 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375394 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 04:03:39.376236 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375397 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 04:03:39.376236 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375399 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 04:03:39.376236 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375403 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 04:03:39.376236 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375405 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 04:03:39.376236 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375408 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 04:03:39.376236 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375412 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 04:03:39.376236 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375417 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 04:03:39.376236 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375420 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 04:03:39.376717 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375422 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 04:03:39.376717 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375425 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 04:03:39.376717 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375429 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 04:03:39.376717 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375432 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 04:03:39.376717 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375435 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 04:03:39.376717 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375439 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 04:03:39.376717 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375442 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 04:03:39.376717 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375445 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 04:03:39.376717 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375448 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 04:03:39.376717 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375450 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 04:03:39.376717 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375453 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 04:03:39.376717 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375455 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 04:03:39.376717 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375458 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 04:03:39.376717 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375461 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 04:03:39.376717 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375463 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 04:03:39.376717 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375466 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 04:03:39.376717 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375469 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 04:03:39.376717 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375471 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 04:03:39.376717 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375474 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 04:03:39.377184 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375478 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 04:03:39.377184 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375482 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 04:03:39.377184 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375484 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 04:03:39.377184 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375487 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 04:03:39.377184 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375490 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 04:03:39.377184 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375492 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 04:03:39.377184 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375496 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 04:03:39.377184 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375499 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 04:03:39.377184 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375501 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 04:03:39.377184 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375504 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 04:03:39.377184 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375506 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 04:03:39.377184 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375509 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 04:03:39.377184 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375512 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 04:03:39.377184 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375515 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 04:03:39.377184 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375518 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 04:03:39.377184 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375520 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 04:03:39.377184 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375523 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 04:03:39.377184 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375525 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 04:03:39.377184 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375528 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 04:03:39.377184 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375531 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 04:03:39.377685 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375533 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 04:03:39.377685 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375537 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 04:03:39.377685 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375540 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 04:03:39.377685 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375542 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 04:03:39.377685 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375545 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 04:03:39.377685 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375548 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 04:03:39.377685 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375551 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 04:03:39.377685 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375553 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 04:03:39.377685 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375556 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 29 04:03:39.377685 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375559 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 04:03:39.377685 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375562 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 04:03:39.377685 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375564 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 04:03:39.377685 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375567 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 04:03:39.377685 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.375569 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 04:03:39.377685 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376282 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 04:03:39.377685 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376290 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 04:03:39.377685 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376293 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 04:03:39.377685 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376296 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 04:03:39.377685 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376298 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 04:03:39.377685 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376301 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 04:03:39.378164 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376304 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 04:03:39.378164 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376309 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 04:03:39.378164 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376312 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 04:03:39.378164 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376315 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 04:03:39.378164 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376318 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 04:03:39.378164 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376321 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 04:03:39.378164 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376324 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 04:03:39.378164 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376327 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 04:03:39.378164 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376330 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 04:03:39.378164 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376333 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 04:03:39.378164 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376336 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 04:03:39.378164 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376339 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 04:03:39.378164 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376342 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 04:03:39.378164 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376344 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 04:03:39.378164 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376348 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 04:03:39.378164 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376350 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 04:03:39.378164 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376353 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 04:03:39.378164 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376355 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 04:03:39.378164 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376357 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 04:03:39.378644 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376360 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 04:03:39.378644 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376363 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 04:03:39.378644 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376365 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 04:03:39.378644 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376368 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 04:03:39.378644 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376370 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 04:03:39.378644 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376373 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 04:03:39.378644 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376375 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 04:03:39.378644 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376378 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 04:03:39.378644 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376380 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 04:03:39.378644 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376384 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 04:03:39.378644 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376388 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 04:03:39.378644 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376391 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 04:03:39.378644 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376394 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 04:03:39.378644 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376397 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 04:03:39.378644 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376399 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 04:03:39.378644 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376402 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 04:03:39.378644 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376404 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 04:03:39.378644 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376407 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 04:03:39.378644 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376410 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 04:03:39.378644 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376412 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 04:03:39.379146 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376415 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 04:03:39.379146 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376418 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 04:03:39.379146 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376420 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 04:03:39.379146 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376422 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 04:03:39.379146 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376426 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 04:03:39.379146 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376429 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 04:03:39.379146 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376432 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 04:03:39.379146 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376434 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 04:03:39.379146 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376437 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 04:03:39.379146 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376439 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 04:03:39.379146 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376442 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 04:03:39.379146 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376444 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 04:03:39.379146 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376447 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 04:03:39.379146 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376450 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 04:03:39.379146 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376452 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 04:03:39.379146 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376454 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 04:03:39.379146 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376457 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 04:03:39.379146 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376459 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 04:03:39.379146 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376462 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 04:03:39.379146 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376464 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 04:03:39.379642 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376467 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 04:03:39.379642 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376469 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 04:03:39.379642 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376473 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 04:03:39.379642 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376476 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 04:03:39.379642 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376479 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 04:03:39.379642 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376481 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 04:03:39.379642 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376484 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 04:03:39.379642 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376486 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 04:03:39.379642 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376489 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 04:03:39.379642 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376492 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 04:03:39.379642 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376494 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 04:03:39.379642 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376496 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 04:03:39.379642 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376499 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 04:03:39.379642 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376501 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 04:03:39.379642 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376504 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 04:03:39.379642 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376507 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 04:03:39.379642 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376509 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 04:03:39.379642 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376513 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 04:03:39.379642 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376516 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 04:03:39.379642 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376519 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 04:03:39.380159 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376521 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 04:03:39.380159 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376525 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 04:03:39.380159 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376528 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 04:03:39.380159 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376532 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 04:03:39.380159 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376534 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 04:03:39.380159 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376537 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 04:03:39.380159 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376540 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 04:03:39.380159 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376543 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 04:03:39.380159 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376545 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 04:03:39.380159 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376548 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 04:03:39.380159 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376551 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 04:03:39.380159 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376553 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 29 04:03:39.380159 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376556 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 04:03:39.380159 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376561 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 04:03:39.380159 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376935 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 04:03:39.380159 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376939 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 04:03:39.380159 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376942 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 04:03:39.380159 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376945 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 04:03:39.380159 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376947 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 04:03:39.380643 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376950 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 04:03:39.380643 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376952 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 04:03:39.380643 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376955 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 04:03:39.380643 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376957 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 04:03:39.380643 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376960 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 04:03:39.380643 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376963 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 04:03:39.380643 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376965 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 04:03:39.380643 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376967 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 04:03:39.380643 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376970 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 04:03:39.380643 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376973 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 04:03:39.380643 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376975 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 04:03:39.380643 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376978 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 04:03:39.380643 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376981 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 04:03:39.380643 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376983 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 04:03:39.380643 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376986 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 04:03:39.380643 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376988 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 04:03:39.380643 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376991 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 04:03:39.380643 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376993 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 04:03:39.380643 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376996 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 04:03:39.380643 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.376999 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 04:03:39.381139 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377001 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 04:03:39.381139 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377005 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 04:03:39.381139 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377009 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 04:03:39.381139 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377012 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 04:03:39.381139 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377015 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 04:03:39.381139 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377018 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 04:03:39.381139 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377022 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 04:03:39.381139 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377025 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 04:03:39.381139 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377027 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 04:03:39.381139 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377031 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 04:03:39.381139 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377035 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 04:03:39.381139 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377037 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 04:03:39.381139 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377040 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 04:03:39.381139 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377043 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 04:03:39.381139 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377045 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 04:03:39.381139 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377048 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 04:03:39.381139 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377050 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 04:03:39.381139 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377053 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 04:03:39.381634 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377055 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 04:03:39.381634 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377058 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 04:03:39.381634 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377060 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 04:03:39.381634 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377063 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 04:03:39.381634 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377066 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 04:03:39.381634 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377068 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 04:03:39.381634 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377071 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 04:03:39.381634 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377073 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 04:03:39.381634 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377076 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 04:03:39.381634 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377078 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 04:03:39.381634 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377081 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 04:03:39.381634 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377083 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 04:03:39.381634 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377086 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 04:03:39.381634 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377088 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 04:03:39.381634 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377091 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 04:03:39.381634 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377093 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 04:03:39.381634 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377096 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 04:03:39.381634 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377098 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 04:03:39.381634 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377100 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 04:03:39.381634 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377103 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 29 04:03:39.382106 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377109 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 04:03:39.382106 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377112 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 04:03:39.382106 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377115 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 04:03:39.382106 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377118 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 04:03:39.382106 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377121 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 04:03:39.382106 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377124 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 04:03:39.382106 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377127 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 04:03:39.382106 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377129 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 04:03:39.382106 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377131 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 04:03:39.382106 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377134 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 04:03:39.382106 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377136 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 04:03:39.382106 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377139 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 04:03:39.382106 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377143 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 04:03:39.382106 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377145 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 04:03:39.382106 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377148 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 04:03:39.382106 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377150 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 04:03:39.382106 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377153 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 04:03:39.382106 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377156 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 04:03:39.382106 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377158 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 04:03:39.382106 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377161 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 04:03:39.382604 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377164 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 04:03:39.382604 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377167 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 04:03:39.382604 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377169 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 04:03:39.382604 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377172 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 04:03:39.382604 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377174 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 04:03:39.382604 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377177 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 04:03:39.382604 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377179 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 04:03:39.382604 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377181 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 04:03:39.382604 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377184 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 04:03:39.382604 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377186 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 04:03:39.382604 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377189 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 04:03:39.382604 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377191 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 04:03:39.382604 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377195 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 04:03:39.382604 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377198 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 04:03:39.382604 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377201 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 04:03:39.382604 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377217 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 04:03:39.382604 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377293 2579 flags.go:64] FLAG: --address="0.0.0.0" Jul 29 04:03:39.382604 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377301 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 29 04:03:39.382604 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377308 2579 flags.go:64] FLAG: --anonymous-auth="true" Jul 29 04:03:39.382604 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377312 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 29 04:03:39.382604 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377316 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 29 04:03:39.383147 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377319 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 29 04:03:39.383147 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377323 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 29 04:03:39.383147 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377328 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 29 04:03:39.383147 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377331 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 29 04:03:39.383147 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377333 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 29 04:03:39.383147 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377337 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 29 04:03:39.383147 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377340 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 29 04:03:39.383147 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377343 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 29 04:03:39.383147 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377348 2579 flags.go:64] FLAG: --cgroup-root="" Jul 29 04:03:39.383147 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377351 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 29 04:03:39.383147 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377354 2579 flags.go:64] FLAG: --client-ca-file="" Jul 29 04:03:39.383147 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377357 2579 flags.go:64] FLAG: --cloud-provider="external" Jul 29 04:03:39.383147 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377361 2579 flags.go:64] FLAG: --cluster-dns="[]" Jul 29 04:03:39.383147 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377365 2579 flags.go:64] FLAG: --cluster-domain="" Jul 29 04:03:39.383147 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377368 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 29 04:03:39.383147 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377371 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 29 04:03:39.383147 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377374 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 29 04:03:39.383147 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377377 2579 flags.go:64] FLAG: --container-log-max-files="5" Jul 29 04:03:39.383147 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377381 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 29 04:03:39.383147 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377384 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 29 04:03:39.383147 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377386 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 29 04:03:39.383147 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377390 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 29 04:03:39.383147 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377393 2579 flags.go:64] FLAG: --contention-profiling="false" Jul 29 04:03:39.383147 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377397 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 29 04:03:39.383731 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377400 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 29 04:03:39.383731 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377403 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 29 04:03:39.383731 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377405 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 29 04:03:39.383731 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377409 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 29 04:03:39.383731 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377412 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 29 04:03:39.383731 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377415 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 29 04:03:39.383731 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377418 2579 flags.go:64] FLAG: --enable-load-reader="false" Jul 29 04:03:39.383731 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377421 2579 flags.go:64] FLAG: --enable-server="true" Jul 29 04:03:39.383731 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377424 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 29 04:03:39.383731 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377428 2579 flags.go:64] FLAG: --event-burst="100" Jul 29 04:03:39.383731 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377431 2579 flags.go:64] FLAG: --event-qps="50" Jul 29 04:03:39.383731 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377434 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 29 04:03:39.383731 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377437 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 29 04:03:39.383731 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377439 2579 flags.go:64] FLAG: --eviction-hard="" Jul 29 04:03:39.383731 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377443 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 29 04:03:39.383731 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377446 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 29 04:03:39.383731 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377452 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 29 04:03:39.383731 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377455 2579 flags.go:64] FLAG: --eviction-soft="" Jul 29 04:03:39.383731 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377458 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 29 04:03:39.383731 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377461 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 29 04:03:39.383731 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377465 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 29 04:03:39.383731 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377468 2579 flags.go:64] FLAG: --experimental-mounter-path="" Jul 29 04:03:39.383731 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377471 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 29 04:03:39.383731 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377474 2579 flags.go:64] FLAG: --fail-swap-on="true" Jul 29 04:03:39.383731 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377476 2579 flags.go:64] FLAG: --feature-gates="" Jul 29 04:03:39.384385 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377480 2579 flags.go:64] FLAG: --file-check-frequency="20s" Jul 29 04:03:39.384385 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377483 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 29 04:03:39.384385 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377486 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 29 04:03:39.384385 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377489 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 29 04:03:39.384385 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377492 2579 flags.go:64] FLAG: --healthz-port="10248" Jul 29 04:03:39.384385 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377495 2579 flags.go:64] FLAG: --help="false" Jul 29 04:03:39.384385 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377499 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-139-111.ec2.internal" Jul 29 04:03:39.384385 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377502 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 29 04:03:39.384385 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377505 2579 flags.go:64] FLAG: --http-check-frequency="20s" Jul 29 04:03:39.384385 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377508 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 29 04:03:39.384385 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377511 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 29 04:03:39.384385 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377515 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 29 04:03:39.384385 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377518 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 29 04:03:39.384385 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377520 2579 flags.go:64] FLAG: --image-service-endpoint="" Jul 29 04:03:39.384385 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377523 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 29 04:03:39.384385 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377526 2579 flags.go:64] FLAG: --kube-api-burst="100" Jul 29 04:03:39.384385 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377529 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 29 04:03:39.384385 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377532 2579 flags.go:64] FLAG: --kube-api-qps="50" Jul 29 04:03:39.384385 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377535 2579 flags.go:64] FLAG: --kube-reserved="" Jul 29 04:03:39.384385 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377537 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 29 04:03:39.384385 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377540 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 29 04:03:39.384385 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377543 2579 flags.go:64] FLAG: --kubelet-cgroups="" Jul 29 04:03:39.384385 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377546 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 29 04:03:39.384385 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377550 2579 flags.go:64] FLAG: --lock-file="" Jul 29 04:03:39.384973 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377552 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 29 04:03:39.384973 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377555 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 29 04:03:39.384973 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377558 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 29 04:03:39.384973 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377563 2579 flags.go:64] FLAG: --log-json-split-stream="false" Jul 29 04:03:39.384973 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377567 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 29 04:03:39.384973 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377569 2579 flags.go:64] FLAG: --log-text-split-stream="false" Jul 29 04:03:39.384973 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377572 2579 flags.go:64] FLAG: --logging-format="text" Jul 29 04:03:39.384973 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377575 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 29 04:03:39.384973 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377578 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 29 04:03:39.384973 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377581 2579 flags.go:64] FLAG: --manifest-url="" Jul 29 04:03:39.384973 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377584 2579 flags.go:64] FLAG: --manifest-url-header="" Jul 29 04:03:39.384973 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377588 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 29 04:03:39.384973 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377591 2579 flags.go:64] FLAG: --max-open-files="1000000" Jul 29 04:03:39.384973 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377595 2579 flags.go:64] FLAG: --max-pods="110" Jul 29 04:03:39.384973 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377599 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 29 04:03:39.384973 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377602 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 29 04:03:39.384973 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377605 2579 flags.go:64] FLAG: --memory-manager-policy="None" Jul 29 04:03:39.384973 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377608 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 29 04:03:39.384973 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377611 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 29 04:03:39.384973 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377613 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 29 04:03:39.384973 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377616 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 29 04:03:39.384973 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377623 2579 flags.go:64] FLAG: --node-status-max-images="50" Jul 29 04:03:39.384973 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377626 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 29 04:03:39.384973 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377629 2579 flags.go:64] FLAG: --oom-score-adj="-999" Jul 29 04:03:39.384973 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377633 2579 flags.go:64] FLAG: --pod-cidr="" Jul 29 04:03:39.385593 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377636 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 29 04:03:39.385593 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377641 2579 flags.go:64] FLAG: --pod-manifest-path="" Jul 29 04:03:39.385593 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377644 2579 flags.go:64] FLAG: --pod-max-pids="-1" Jul 29 04:03:39.385593 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377647 2579 flags.go:64] FLAG: --pods-per-core="0" Jul 29 04:03:39.385593 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377650 2579 flags.go:64] FLAG: --port="10250" Jul 29 04:03:39.385593 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377653 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 29 04:03:39.385593 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377655 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0171eaeddb7c221b3" Jul 29 04:03:39.385593 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377660 2579 flags.go:64] FLAG: --qos-reserved="" Jul 29 04:03:39.385593 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377663 2579 flags.go:64] FLAG: --read-only-port="10255" Jul 29 04:03:39.385593 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377666 2579 flags.go:64] FLAG: --register-node="true" Jul 29 04:03:39.385593 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377669 2579 flags.go:64] FLAG: --register-with-taints="" Jul 29 04:03:39.385593 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377673 2579 flags.go:64] FLAG: --registry-burst="10" Jul 29 04:03:39.385593 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377676 2579 flags.go:64] FLAG: --registry-qps="5" Jul 29 04:03:39.385593 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377679 2579 flags.go:64] FLAG: --reserved-cpus="" Jul 29 04:03:39.385593 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377682 2579 flags.go:64] FLAG: --reserved-memory="" Jul 29 04:03:39.385593 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377685 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 29 04:03:39.385593 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377688 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 29 04:03:39.385593 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377692 2579 flags.go:64] FLAG: --rotate-certificates="false" Jul 29 04:03:39.385593 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377695 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 29 04:03:39.385593 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377698 2579 flags.go:64] FLAG: --runonce="false" Jul 29 04:03:39.385593 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377701 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 29 04:03:39.385593 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377704 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 29 04:03:39.385593 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377707 2579 flags.go:64] FLAG: --seccomp-default="false" Jul 29 04:03:39.385593 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377710 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 29 04:03:39.385593 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377713 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 29 04:03:39.386260 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377716 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 29 04:03:39.386260 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377719 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 29 04:03:39.386260 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377722 2579 flags.go:64] FLAG: --storage-driver-password="root" Jul 29 04:03:39.386260 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377725 2579 flags.go:64] FLAG: --storage-driver-secure="false" Jul 29 04:03:39.386260 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377729 2579 flags.go:64] FLAG: --storage-driver-table="stats" Jul 29 04:03:39.386260 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377732 2579 flags.go:64] FLAG: --storage-driver-user="root" Jul 29 04:03:39.386260 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377734 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 29 04:03:39.386260 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377737 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 29 04:03:39.386260 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377740 2579 flags.go:64] FLAG: --system-cgroups="" Jul 29 04:03:39.386260 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377743 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 29 04:03:39.386260 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377748 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 29 04:03:39.386260 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377751 2579 flags.go:64] FLAG: --tls-cert-file="" Jul 29 04:03:39.386260 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377754 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 29 04:03:39.386260 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377757 2579 flags.go:64] FLAG: --tls-min-version="" Jul 29 04:03:39.386260 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377761 2579 flags.go:64] FLAG: --tls-private-key-file="" Jul 29 04:03:39.386260 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377764 2579 flags.go:64] FLAG: --topology-manager-policy="none" Jul 29 04:03:39.386260 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377767 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 29 04:03:39.386260 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377770 2579 flags.go:64] FLAG: --topology-manager-scope="container" Jul 29 04:03:39.386260 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377774 2579 flags.go:64] FLAG: --v="2" Jul 29 04:03:39.386260 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377778 2579 flags.go:64] FLAG: --version="false" Jul 29 04:03:39.386260 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377783 2579 flags.go:64] FLAG: --vmodule="" Jul 29 04:03:39.386260 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377786 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 29 04:03:39.386260 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.377789 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 29 04:03:39.386260 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377904 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 04:03:39.386260 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377908 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 04:03:39.386875 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377911 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 04:03:39.386875 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377914 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 04:03:39.386875 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377917 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 04:03:39.386875 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377920 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 04:03:39.386875 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377942 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 04:03:39.386875 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377948 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 04:03:39.386875 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377951 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 04:03:39.386875 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377954 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 04:03:39.386875 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377957 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 04:03:39.386875 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377960 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 04:03:39.386875 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377963 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 04:03:39.386875 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377966 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 04:03:39.386875 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377969 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 04:03:39.386875 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377971 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 04:03:39.386875 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377974 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 04:03:39.386875 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377977 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 04:03:39.386875 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377979 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 04:03:39.386875 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377982 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 04:03:39.386875 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377985 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 04:03:39.386875 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377987 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 04:03:39.387387 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377990 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 04:03:39.387387 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377994 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 04:03:39.387387 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377996 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 04:03:39.387387 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.377999 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 04:03:39.387387 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378002 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 04:03:39.387387 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378005 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 04:03:39.387387 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378007 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 04:03:39.387387 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378010 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 04:03:39.387387 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378015 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 04:03:39.387387 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378019 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 04:03:39.387387 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378022 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 04:03:39.387387 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378025 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 04:03:39.387387 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378028 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 04:03:39.387387 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378031 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 04:03:39.387387 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378034 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 04:03:39.387387 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378036 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 04:03:39.387387 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378039 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 04:03:39.387387 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378042 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 04:03:39.387387 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378045 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 04:03:39.387387 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378048 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 04:03:39.387880 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378050 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 04:03:39.387880 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378053 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 04:03:39.387880 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378056 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 04:03:39.387880 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378059 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 04:03:39.387880 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378062 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 04:03:39.387880 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378065 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 04:03:39.387880 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378068 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 04:03:39.387880 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378070 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 04:03:39.387880 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378073 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 04:03:39.387880 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378076 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 04:03:39.387880 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378078 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 04:03:39.387880 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378081 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 04:03:39.387880 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378083 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 04:03:39.387880 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378087 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 04:03:39.387880 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378090 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 04:03:39.387880 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378092 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 04:03:39.387880 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378095 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 04:03:39.387880 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378098 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 04:03:39.387880 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378101 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 04:03:39.388354 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378104 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 04:03:39.388354 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378106 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 04:03:39.388354 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378109 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 04:03:39.388354 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378112 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 04:03:39.388354 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378114 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 04:03:39.388354 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378117 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 04:03:39.388354 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378120 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 04:03:39.388354 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378122 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 04:03:39.388354 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378125 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 04:03:39.388354 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378128 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 04:03:39.388354 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378131 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 04:03:39.388354 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378133 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 04:03:39.388354 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378135 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 04:03:39.388354 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378138 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 04:03:39.388354 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378142 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 04:03:39.388354 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378145 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 04:03:39.388354 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378148 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 04:03:39.388354 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378151 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 04:03:39.388354 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378154 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 04:03:39.388828 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378156 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 04:03:39.388828 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378159 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 04:03:39.388828 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378162 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 29 04:03:39.388828 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378164 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 04:03:39.388828 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378178 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 04:03:39.388828 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378182 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 04:03:39.388828 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378185 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 04:03:39.388828 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378190 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 04:03:39.388828 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378192 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 04:03:39.388828 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378195 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 04:03:39.388828 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378199 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 04:03:39.388828 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378213 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 04:03:39.388828 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378216 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 04:03:39.388828 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378219 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 04:03:39.388828 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378221 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 04:03:39.388828 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378224 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 04:03:39.388828 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378227 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 04:03:39.388828 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378230 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 04:03:39.388828 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.378233 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 04:03:39.389402 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.378238 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 04:03:39.389402 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.385158 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 29 04:03:39.389402 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.385173 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 29 04:03:39.389402 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.385795 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 29 04:03:39.389402 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.385802 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 29 04:03:39.389402 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385864 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 04:03:39.389402 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385868 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 04:03:39.389402 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385872 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 04:03:39.389402 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385875 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 04:03:39.389402 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385878 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 04:03:39.389402 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385881 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 04:03:39.389402 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385884 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 04:03:39.389402 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385887 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 04:03:39.389402 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385890 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 04:03:39.389402 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385893 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 04:03:39.389772 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385896 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 04:03:39.389772 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385898 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 04:03:39.389772 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385901 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 04:03:39.389772 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385903 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 04:03:39.389772 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385906 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 04:03:39.389772 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385909 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 04:03:39.389772 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385912 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 04:03:39.389772 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385914 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 04:03:39.389772 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385918 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 04:03:39.389772 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385922 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 04:03:39.389772 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385926 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 04:03:39.389772 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385930 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 04:03:39.389772 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385933 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 04:03:39.389772 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385937 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 04:03:39.389772 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385940 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 04:03:39.389772 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385943 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 04:03:39.389772 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385946 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 04:03:39.389772 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385948 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 04:03:39.389772 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385951 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 04:03:39.390350 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385955 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 04:03:39.390350 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385958 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 04:03:39.390350 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385960 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 04:03:39.390350 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385963 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 04:03:39.390350 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385966 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 29 04:03:39.390350 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385968 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 04:03:39.390350 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385971 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 04:03:39.390350 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385973 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 04:03:39.390350 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385976 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 04:03:39.390350 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385979 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 04:03:39.390350 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385981 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 04:03:39.390350 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385983 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 04:03:39.390350 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385986 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 04:03:39.390350 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385989 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 04:03:39.390350 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385991 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 04:03:39.390350 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385993 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 04:03:39.390350 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385996 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 04:03:39.390350 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.385999 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 04:03:39.390350 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386002 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 04:03:39.390817 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386005 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 04:03:39.390817 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386007 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 04:03:39.390817 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386010 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 04:03:39.390817 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386013 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 04:03:39.390817 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386015 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 04:03:39.390817 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386018 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 04:03:39.390817 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386021 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 04:03:39.390817 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386023 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 04:03:39.390817 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386026 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 04:03:39.390817 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386029 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 04:03:39.390817 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386031 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 04:03:39.390817 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386034 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 04:03:39.390817 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386036 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 04:03:39.390817 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386040 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 04:03:39.390817 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386042 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 04:03:39.390817 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386045 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 04:03:39.390817 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386047 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 04:03:39.390817 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386050 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 04:03:39.390817 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386052 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 04:03:39.390817 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386055 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 04:03:39.391324 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386057 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 04:03:39.391324 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386060 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 04:03:39.391324 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386062 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 04:03:39.391324 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386065 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 04:03:39.391324 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386068 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 04:03:39.391324 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386070 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 04:03:39.391324 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386073 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 04:03:39.391324 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386075 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 04:03:39.391324 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386078 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 04:03:39.391324 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386080 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 04:03:39.391324 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386084 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 04:03:39.391324 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386087 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 04:03:39.391324 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386090 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 04:03:39.391324 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386093 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 04:03:39.391324 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386095 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 04:03:39.391324 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386098 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 04:03:39.391324 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386100 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 04:03:39.391324 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386103 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 04:03:39.391324 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386105 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 04:03:39.391324 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386108 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 04:03:39.391811 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386110 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 04:03:39.391811 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386112 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 04:03:39.391811 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386115 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 04:03:39.391811 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386118 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 04:03:39.391811 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386121 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 04:03:39.391811 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386124 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 04:03:39.391811 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386127 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 04:03:39.391811 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386129 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 04:03:39.391811 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386132 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 04:03:39.391811 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386134 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 04:03:39.391811 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386137 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 04:03:39.391811 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.386142 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 04:03:39.391811 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386304 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 04:03:39.391811 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386310 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 04:03:39.391811 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386313 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 04:03:39.392260 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386316 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 04:03:39.392260 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386319 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 04:03:39.392260 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386321 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 04:03:39.392260 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386324 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 04:03:39.392260 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386327 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 04:03:39.392260 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386329 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 04:03:39.392260 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386332 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 04:03:39.392260 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386335 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 04:03:39.392260 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386338 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 04:03:39.392260 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386342 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 04:03:39.392260 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386345 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 04:03:39.392260 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386348 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 04:03:39.392260 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386350 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 04:03:39.392260 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386353 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 04:03:39.392260 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386356 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 04:03:39.392260 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386358 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 04:03:39.392260 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386361 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 04:03:39.392260 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386364 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 04:03:39.392260 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386367 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 04:03:39.392260 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386369 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 04:03:39.392736 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386373 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 04:03:39.392736 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386375 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 04:03:39.392736 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386378 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 04:03:39.392736 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386380 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 04:03:39.392736 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386383 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 04:03:39.392736 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386385 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 04:03:39.392736 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386388 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 04:03:39.392736 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386390 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 04:03:39.392736 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386392 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 04:03:39.392736 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386395 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 04:03:39.392736 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386398 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 04:03:39.392736 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386401 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 04:03:39.392736 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386405 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 04:03:39.392736 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386407 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 29 04:03:39.392736 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386410 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 04:03:39.392736 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386413 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 04:03:39.392736 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386415 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 04:03:39.392736 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386418 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 04:03:39.392736 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386420 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 04:03:39.393216 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386423 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 04:03:39.393216 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386425 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 04:03:39.393216 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386429 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 04:03:39.393216 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386432 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 04:03:39.393216 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386435 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 04:03:39.393216 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386438 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 04:03:39.393216 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386440 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 04:03:39.393216 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386443 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 04:03:39.393216 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386446 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 04:03:39.393216 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386448 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 04:03:39.393216 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386451 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 04:03:39.393216 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386453 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 04:03:39.393216 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386456 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 04:03:39.393216 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386458 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 04:03:39.393216 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386462 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 04:03:39.393216 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386465 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 04:03:39.393216 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386467 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 04:03:39.393216 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386470 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 04:03:39.393216 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386473 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 04:03:39.393692 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386475 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 04:03:39.393692 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386477 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 04:03:39.393692 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386480 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 04:03:39.393692 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386482 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 04:03:39.393692 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386485 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 04:03:39.393692 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386487 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 04:03:39.393692 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386490 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 04:03:39.393692 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386492 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 04:03:39.393692 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386495 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 04:03:39.393692 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386497 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 04:03:39.393692 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386500 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 04:03:39.393692 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386504 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 04:03:39.393692 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386507 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 04:03:39.393692 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386510 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 04:03:39.393692 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386512 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 04:03:39.393692 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386515 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 04:03:39.393692 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386517 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 04:03:39.393692 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386520 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 04:03:39.393692 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386523 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 04:03:39.393692 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386525 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 04:03:39.394174 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386528 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 04:03:39.394174 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386530 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 04:03:39.394174 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386533 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 04:03:39.394174 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386535 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 04:03:39.394174 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386538 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 04:03:39.394174 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386541 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 04:03:39.394174 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386543 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 04:03:39.394174 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386545 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 04:03:39.394174 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386548 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 04:03:39.394174 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386551 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 04:03:39.394174 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386554 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 04:03:39.394174 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386556 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 04:03:39.394174 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386558 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 04:03:39.394174 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386561 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 04:03:39.394174 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386564 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 04:03:39.394174 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386566 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 04:03:39.394174 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386569 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 04:03:39.394174 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:39.386572 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 04:03:39.394703 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.386577 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 04:03:39.394703 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.386682 2579 server.go:962] "Client rotation is on, will bootstrap in background" Jul 29 04:03:39.394703 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.388707 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 29 04:03:39.394703 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.389967 2579 server.go:1019] "Starting client certificate rotation" Jul 29 04:03:39.394703 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.390066 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 04:03:39.394703 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.390107 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 04:03:39.394703 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.392159 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 04:03:39.395138 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.395124 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 04:03:39.406808 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.406792 2579 log.go:25] "Validated CRI v1 runtime API" Jul 29 04:03:39.409674 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.409649 2579 log.go:25] "Validated CRI v1 image API" Jul 29 04:03:39.410992 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.410972 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 29 04:03:39.413089 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.413063 2579 fs.go:135] Filesystem UUIDs: map[3122850a-22c6-490b-97a3-669d7a390bd9:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c0d4c4de-5c21-4f1e-8283-259a70593abd:/dev/nvme0n1p4] Jul 29 04:03:39.413171 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.413086 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 29 04:03:39.417461 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.417331 2579 manager.go:216] Machine: {Timestamp:2026-07-29 04:03:39.416811019 +0000 UTC m=+0.276185733 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3120629 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2903fd6bf6fc1560a70a8624a8d4dd SystemUUID:ec2903fd-6bf6-fc15-60a7-0a8624a8d4dd BootID:cb2ee790-f3d8-49b5-b0a0-928d46c8af11 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:88:74:3c:8b:83 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:88:74:3c:8b:83 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:7f:52:08:d2:5a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 29 04:03:39.417461 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.417442 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 29 04:03:39.417627 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.417603 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 29 04:03:39.419062 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.419042 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 04:03:39.420260 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.420231 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 29 04:03:39.420431 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.420267 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-111.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 29 04:03:39.420950 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.420938 2579 topology_manager.go:138] "Creating topology manager with none policy" Jul 29 04:03:39.421007 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.420957 2579 container_manager_linux.go:309] "Creating device plugin manager" Jul 29 04:03:39.421007 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.420968 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 04:03:39.422583 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.422571 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 04:03:39.423387 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.423375 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 29 04:03:39.426497 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.426484 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 29 04:03:39.426634 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.426624 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 29 04:03:39.426691 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.426660 2579 kubelet.go:486] "Attempting to sync node with API server" Jul 29 04:03:39.426691 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.426674 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 29 04:03:39.426691 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.426689 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 29 04:03:39.426810 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.426703 2579 kubelet.go:398] "Adding apiserver pod source" Jul 29 04:03:39.426810 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.426714 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 29 04:03:39.427115 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.427100 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 04:03:39.427187 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.427124 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 04:03:39.428949 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.428932 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 29 04:03:39.430765 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.430750 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 29 04:03:39.430817 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.430780 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 29 04:03:39.432315 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.432301 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 29 04:03:39.432360 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.432324 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 29 04:03:39.432360 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.432331 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 29 04:03:39.432360 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.432336 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 29 04:03:39.432360 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.432344 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 29 04:03:39.432360 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.432349 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 29 04:03:39.432360 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.432357 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 29 04:03:39.432360 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.432362 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 29 04:03:39.432534 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.432370 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 29 04:03:39.432534 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.432379 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 29 04:03:39.432534 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.432390 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 29 04:03:39.432534 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.432396 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 29 04:03:39.432968 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.432960 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 29 04:03:39.433000 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.432969 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 29 04:03:39.434553 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.434526 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gmx8f" Jul 29 04:03:39.435849 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.435834 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-111.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 29 04:03:39.436056 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:39.436026 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-111.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 29 04:03:39.436112 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:39.436034 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 29 04:03:39.436236 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.436224 2579 server.go:1268] "Started kubelet" Jul 29 04:03:39.436348 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.436319 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 29 04:03:39.436348 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.436310 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 29 04:03:39.436449 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.436375 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 29 04:03:39.437096 ip-10-0-139-111 systemd[1]: Started Kubernetes Kubelet. Jul 29 04:03:39.438670 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.438653 2579 server.go:310] "Adding debug handlers to kubelet server" Jul 29 04:03:39.440726 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.440708 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 29 04:03:39.440924 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:39.440155 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-111.ec2.internal.18c6a6e869c5edf8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-111.ec2.internal,UID:ip-10-0-139-111.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-111.ec2.internal,},FirstTimestamp:2026-07-29 04:03:39.436183032 +0000 UTC m=+0.295557749,LastTimestamp:2026-07-29 04:03:39.436183032 +0000 UTC m=+0.295557749,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-111.ec2.internal,}" Jul 29 04:03:39.441512 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:39.441493 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 29 04:03:39.441692 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.441680 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gmx8f" Jul 29 04:03:39.447945 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.447926 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 29 04:03:39.447945 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.447929 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 29 04:03:39.448101 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:39.448037 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-111.ec2.internal\" not found" Jul 29 04:03:39.448853 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.448714 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 29 04:03:39.448903 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.448731 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 29 04:03:39.448966 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.448951 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 29 04:03:39.449028 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.448995 2579 factory.go:55] Registering systemd factory Jul 29 04:03:39.449091 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.449082 2579 factory.go:223] Registration of the systemd container factory successfully Jul 29 04:03:39.449150 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.449137 2579 reconstruct.go:97] "Volume reconstruction finished" Jul 29 04:03:39.449191 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.449151 2579 reconciler.go:29] "Reconciler: start to sync state" Jul 29 04:03:39.449479 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.449459 2579 factory.go:153] Registering CRI-O factory Jul 29 04:03:39.449479 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.449478 2579 factory.go:223] Registration of the crio container factory successfully Jul 29 04:03:39.449575 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.449529 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 29 04:03:39.449575 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.449554 2579 factory.go:103] Registering Raw factory Jul 29 04:03:39.449575 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.449569 2579 manager.go:1196] Started watching for new ooms in manager Jul 29 04:03:39.451036 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.451011 2579 manager.go:349] Starting recovery of all containers Jul 29 04:03:39.451194 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:39.451170 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 29 04:03:39.451314 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:39.451289 2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-111.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 29 04:03:39.463128 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.463100 2579 manager.go:354] Recovery completed Jul 29 04:03:39.467234 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.467219 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 04:03:39.469026 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.469008 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-111.ec2.internal" event="NodeHasSufficientMemory" Jul 29 04:03:39.469117 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.469045 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-111.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 04:03:39.469117 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.469060 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-111.ec2.internal" event="NodeHasSufficientPID" Jul 29 04:03:39.469665 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.469652 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 04:03:39.469723 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.469653 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 29 04:03:39.469723 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.469704 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 29 04:03:39.469786 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.469726 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 29 04:03:39.470390 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.470374 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-111.ec2.internal" event="NodeHasSufficientMemory" Jul 29 04:03:39.470476 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.470406 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-111.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 04:03:39.470476 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.470420 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-111.ec2.internal" event="NodeHasSufficientPID" Jul 29 04:03:39.472358 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.472345 2579 policy_none.go:49] "None policy: Start" Jul 29 04:03:39.472405 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.472360 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 29 04:03:39.472405 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.472370 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 29 04:03:39.474372 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.474361 2579 policy_none.go:47] "Start" Jul 29 04:03:39.519914 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.519747 2579 manager.go:337] "Starting Device Plugin manager" Jul 29 04:03:39.519999 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:39.519944 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 29 04:03:39.519999 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.519956 2579 server.go:87] "Starting device plugin registration server" Jul 29 04:03:39.520167 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.520157 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 29 04:03:39.520219 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.520170 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 29 04:03:39.520486 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.520366 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 29 04:03:39.520486 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.520454 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 29 04:03:39.520486 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.520466 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 29 04:03:39.520916 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:39.520894 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 29 04:03:39.520985 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:39.520940 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-111.ec2.internal\" not found" Jul 29 04:03:39.583754 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.583707 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 29 04:03:39.585943 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.584931 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 29 04:03:39.585943 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.584967 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 29 04:03:39.585943 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.584991 2579 kubelet.go:2448] "Starting kubelet main sync loop" Jul 29 04:03:39.585943 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:39.585025 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 29 04:03:39.587764 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.587745 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 04:03:39.621097 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.621036 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 04:03:39.622064 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.622048 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-111.ec2.internal" event="NodeHasSufficientMemory" Jul 29 04:03:39.622133 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.622080 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-111.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 04:03:39.622133 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.622122 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-111.ec2.internal" event="NodeHasSufficientPID" Jul 29 04:03:39.622200 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.622147 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-111.ec2.internal" Jul 29 04:03:39.627914 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.627898 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-111.ec2.internal" Jul 29 04:03:39.627963 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:39.627920 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-111.ec2.internal\": node \"ip-10-0-139-111.ec2.internal\" not found" Jul 29 04:03:39.639499 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:39.639471 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-111.ec2.internal\" not found" Jul 29 04:03:39.685526 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.685500 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-111.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-111.ec2.internal"] Jul 29 04:03:39.685631 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.685602 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 04:03:39.686651 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.686634 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-111.ec2.internal" event="NodeHasSufficientMemory" Jul 29 04:03:39.686715 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.686667 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-111.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 04:03:39.686715 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.686677 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-111.ec2.internal" event="NodeHasSufficientPID" Jul 29 04:03:39.689328 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.689314 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 04:03:39.689495 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.689481 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-111.ec2.internal" Jul 29 04:03:39.689532 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.689514 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 04:03:39.690074 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.690055 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-111.ec2.internal" event="NodeHasSufficientMemory" Jul 29 04:03:39.690152 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.690078 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-111.ec2.internal" event="NodeHasSufficientMemory" Jul 29 04:03:39.690152 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.690100 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-111.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 04:03:39.690152 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.690103 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-111.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 04:03:39.690152 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.690116 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-111.ec2.internal" event="NodeHasSufficientPID" Jul 29 04:03:39.690353 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.690115 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-111.ec2.internal" event="NodeHasSufficientPID" Jul 29 04:03:39.692470 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.692453 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-111.ec2.internal" Jul 29 04:03:39.692560 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.692478 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 04:03:39.693239 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.693222 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-111.ec2.internal" event="NodeHasSufficientMemory" Jul 29 04:03:39.693326 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.693250 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-111.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 04:03:39.693326 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.693279 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-111.ec2.internal" event="NodeHasSufficientPID" Jul 29 04:03:39.724818 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:39.724793 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-111.ec2.internal\" not found" node="ip-10-0-139-111.ec2.internal" Jul 29 04:03:39.730121 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:39.730103 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-111.ec2.internal\" not found" node="ip-10-0-139-111.ec2.internal" Jul 29 04:03:39.740258 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:39.740235 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-111.ec2.internal\" not found" Jul 29 04:03:39.750876 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.750857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5aecb2ab7b558cf2544e36bba77ca9dc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-111.ec2.internal\" (UID: \"5aecb2ab7b558cf2544e36bba77ca9dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-111.ec2.internal" Jul 29 04:03:39.750946 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.750881 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5aecb2ab7b558cf2544e36bba77ca9dc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-111.ec2.internal\" (UID: \"5aecb2ab7b558cf2544e36bba77ca9dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-111.ec2.internal" Jul 29 04:03:39.750946 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.750899 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e386db4858c90ca09ab480d4f37fcab1-config\") pod \"kube-apiserver-proxy-ip-10-0-139-111.ec2.internal\" (UID: \"e386db4858c90ca09ab480d4f37fcab1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-111.ec2.internal" Jul 29 04:03:39.841446 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:39.841414 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-111.ec2.internal\" not found" Jul 29 04:03:39.851535 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.851513 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e386db4858c90ca09ab480d4f37fcab1-config\") pod \"kube-apiserver-proxy-ip-10-0-139-111.ec2.internal\" (UID: \"e386db4858c90ca09ab480d4f37fcab1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-111.ec2.internal" Jul 29 04:03:39.851585 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.851539 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5aecb2ab7b558cf2544e36bba77ca9dc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-111.ec2.internal\" (UID: \"5aecb2ab7b558cf2544e36bba77ca9dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-111.ec2.internal" Jul 29 04:03:39.851585 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.851565 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5aecb2ab7b558cf2544e36bba77ca9dc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-111.ec2.internal\" (UID: \"5aecb2ab7b558cf2544e36bba77ca9dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-111.ec2.internal" Jul 29 04:03:39.851657 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.851607 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5aecb2ab7b558cf2544e36bba77ca9dc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-111.ec2.internal\" (UID: \"5aecb2ab7b558cf2544e36bba77ca9dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-111.ec2.internal" Jul 29 04:03:39.851657 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.851638 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5aecb2ab7b558cf2544e36bba77ca9dc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-111.ec2.internal\" (UID: \"5aecb2ab7b558cf2544e36bba77ca9dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-111.ec2.internal" Jul 29 04:03:39.851720 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:39.851638 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e386db4858c90ca09ab480d4f37fcab1-config\") pod \"kube-apiserver-proxy-ip-10-0-139-111.ec2.internal\" (UID: \"e386db4858c90ca09ab480d4f37fcab1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-111.ec2.internal" Jul 29 04:03:39.941742 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:39.941704 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-111.ec2.internal\" not found" Jul 29 04:03:40.033931 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:40.033888 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-111.ec2.internal" Jul 29 04:03:40.034022 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:40.033921 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-111.ec2.internal" Jul 29 04:03:40.042065 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:40.042045 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-111.ec2.internal\" not found" Jul 29 04:03:40.145437 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:40.145393 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-111.ec2.internal\" not found" Jul 29 04:03:40.245674 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:40.245589 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-111.ec2.internal\" not found" Jul 29 04:03:40.346712 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:40.346681 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-111.ec2.internal\" not found" Jul 29 04:03:40.390059 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:40.390027 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 29 04:03:40.390577 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:40.390192 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 04:03:40.446684 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:40.446625 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 03:58:39 +0000 UTC" deadline="2028-01-22 04:17:28.593258944 +0000 UTC" Jul 29 04:03:40.446684 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:40.446677 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13008h13m48.146587214s" Jul 29 04:03:40.447832 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:40.447804 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-111.ec2.internal\" not found" Jul 29 04:03:40.548969 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:40.548931 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-111.ec2.internal\" not found" Jul 29 04:03:40.618183 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:40.618149 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 04:03:40.624349 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:40.624321 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 04:03:40.640657 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:40.640614 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode386db4858c90ca09ab480d4f37fcab1.slice/crio-bddcc0349211fd187f5f9fc86e7bc6e6c6254ef3501a8fbed6f5f224cf110148 WatchSource:0}: Error finding container bddcc0349211fd187f5f9fc86e7bc6e6c6254ef3501a8fbed6f5f224cf110148: Status 404 returned error can't find the container with id bddcc0349211fd187f5f9fc86e7bc6e6c6254ef3501a8fbed6f5f224cf110148 Jul 29 04:03:40.644999 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:40.644976 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aecb2ab7b558cf2544e36bba77ca9dc.slice/crio-35772bc6030192ec15480eaf98513516bb7e78b4936a3e265b71589e8197af3f WatchSource:0}: Error finding container 35772bc6030192ec15480eaf98513516bb7e78b4936a3e265b71589e8197af3f: Status 404 returned error can't find the container with id 35772bc6030192ec15480eaf98513516bb7e78b4936a3e265b71589e8197af3f Jul 29 04:03:40.646454 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:40.646409 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 04:03:40.648110 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:40.648090 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-111.ec2.internal" Jul 29 04:03:40.658814 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:40.658789 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 04:03:40.659717 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:40.659699 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-111.ec2.internal" Jul 29 04:03:40.668349 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:40.668332 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 04:03:40.899334 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:40.899245 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 04:03:41.427858 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.427816 2579 apiserver.go:52] "Watching apiserver" Jul 29 04:03:41.436014 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.435980 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 29 04:03:41.437759 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.437189 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-2qc5b","kube-system/global-pull-secret-syncer-b9zpd","kube-system/kube-apiserver-proxy-ip-10-0-139-111.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxzz5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-111.ec2.internal","openshift-network-diagnostics/network-check-target-x6vhn","openshift-ovn-kubernetes/ovnkube-node-p7792","kube-system/konnectivity-agent-492cr","openshift-cluster-node-tuning-operator/tuned-tf8wh","openshift-dns/node-resolver-v8d4n","openshift-image-registry/node-ca-wwg6z","openshift-multus/multus-additional-cni-plugins-zjpnt","openshift-multus/multus-snqz4","openshift-multus/network-metrics-daemon-zbrk8"] Jul 29 04:03:41.440260 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.440198 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.443007 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.442712 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 29 04:03:41.443730 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.443461 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 29 04:03:41.443730 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.443455 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 29 04:03:41.443730 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.443518 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ctd6g\"" Jul 29 04:03:41.443730 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.443541 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 29 04:03:41.443730 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.443611 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 29 04:03:41.444050 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.443761 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wwg6z" Jul 29 04:03:41.444050 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.443950 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 29 04:03:41.446665 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.446422 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:03:41.446665 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:41.446509 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbrk8" podUID="9fa6e244-be92-4245-a54e-7aa303802f4b" Jul 29 04:03:41.449109 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.449083 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 29 04:03:41.450918 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.450897 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 29 04:03:41.451218 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.451199 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 29 04:03:41.451589 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.451572 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5vddp\"" Jul 29 04:03:41.452151 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.452132 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 29 04:03:41.456677 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.455625 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxzz5" Jul 29 04:03:41.458783 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.458291 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 29 04:03:41.458783 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.458522 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dvm5p\"" Jul 29 04:03:41.458783 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.458730 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 29 04:03:41.459326 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.459012 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 29 04:03:41.459326 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.459097 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-492cr" Jul 29 04:03:41.459326 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.459120 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 04:03:41.461232 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.460725 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-var-lib-openvswitch\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.461232 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.460765 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7fab7167-d61f-486e-85d2-58004704568e-ovnkube-script-lib\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.461232 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.460793 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp6zc\" (UniqueName: \"kubernetes.io/projected/0055fae1-bd7e-4e87-834c-3a317313ae0c-kube-api-access-dp6zc\") pod \"node-ca-wwg6z\" (UID: \"0055fae1-bd7e-4e87-834c-3a317313ae0c\") " pod="openshift-image-registry/node-ca-wwg6z" Jul 29 04:03:41.461232 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.460830 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2582x\" (UniqueName: \"kubernetes.io/projected/4f863b64-3884-435e-b469-06b94f479e3b-kube-api-access-2582x\") pod \"aws-ebs-csi-driver-node-fxzz5\" (UID: \"4f863b64-3884-435e-b469-06b94f479e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxzz5" Jul 29 04:03:41.461232 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.460859 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-run-ovn\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.461232 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.460884 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-log-socket\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.461232 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.460909 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-host-run-ovn-kubernetes\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.461232 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.460939 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7fab7167-d61f-486e-85d2-58004704568e-env-overrides\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.461232 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.460965 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-host-kubelet\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.461232 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.460988 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-host-run-netns\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.461232 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.461010 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-node-log\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.461232 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.461034 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7fab7167-d61f-486e-85d2-58004704568e-ovn-node-metrics-cert\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.461232 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.461059 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0055fae1-bd7e-4e87-834c-3a317313ae0c-host\") pod \"node-ca-wwg6z\" (UID: \"0055fae1-bd7e-4e87-834c-3a317313ae0c\") " pod="openshift-image-registry/node-ca-wwg6z" Jul 29 04:03:41.461232 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.461144 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m8k8\" (UniqueName: \"kubernetes.io/projected/9fa6e244-be92-4245-a54e-7aa303802f4b-kube-api-access-8m8k8\") pod \"network-metrics-daemon-zbrk8\" (UID: \"9fa6e244-be92-4245-a54e-7aa303802f4b\") " pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:03:41.461232 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.461172 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-host-cni-netd\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.461232 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.461198 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7fab7167-d61f-486e-85d2-58004704568e-ovnkube-config\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.462086 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.461223 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f863b64-3884-435e-b469-06b94f479e3b-sys-fs\") pod \"aws-ebs-csi-driver-node-fxzz5\" (UID: \"4f863b64-3884-435e-b469-06b94f479e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxzz5" Jul 29 04:03:41.462086 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.461248 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-host-slash\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.462086 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.461312 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-etc-openvswitch\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.462086 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.461356 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fa6e244-be92-4245-a54e-7aa303802f4b-metrics-certs\") pod \"network-metrics-daemon-zbrk8\" (UID: \"9fa6e244-be92-4245-a54e-7aa303802f4b\") " pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:03:41.462086 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.461381 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f863b64-3884-435e-b469-06b94f479e3b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fxzz5\" (UID: \"4f863b64-3884-435e-b469-06b94f479e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxzz5" Jul 29 04:03:41.462086 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.461407 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f863b64-3884-435e-b469-06b94f479e3b-socket-dir\") pod \"aws-ebs-csi-driver-node-fxzz5\" (UID: \"4f863b64-3884-435e-b469-06b94f479e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxzz5" Jul 29 04:03:41.462086 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.461432 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f863b64-3884-435e-b469-06b94f479e3b-registration-dir\") pod \"aws-ebs-csi-driver-node-fxzz5\" (UID: \"4f863b64-3884-435e-b469-06b94f479e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxzz5" Jul 29 04:03:41.462086 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.461456 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-host-cni-bin\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.462086 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.461482 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk987\" (UniqueName: \"kubernetes.io/projected/7fab7167-d61f-486e-85d2-58004704568e-kube-api-access-rk987\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.462086 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.461506 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0055fae1-bd7e-4e87-834c-3a317313ae0c-serviceca\") pod \"node-ca-wwg6z\" (UID: \"0055fae1-bd7e-4e87-834c-3a317313ae0c\") " pod="openshift-image-registry/node-ca-wwg6z" Jul 29 04:03:41.462086 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.461535 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f863b64-3884-435e-b469-06b94f479e3b-etc-selinux\") pod \"aws-ebs-csi-driver-node-fxzz5\" (UID: \"4f863b64-3884-435e-b469-06b94f479e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxzz5" Jul 29 04:03:41.462086 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.461561 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9384d535-332c-4320-a56e-a8cccbbbc5e8-agent-certs\") pod \"konnectivity-agent-492cr\" (UID: \"9384d535-332c-4320-a56e-a8cccbbbc5e8\") " pod="kube-system/konnectivity-agent-492cr" Jul 29 04:03:41.462086 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.461589 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-systemd-units\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.462086 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.461621 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f863b64-3884-435e-b469-06b94f479e3b-device-dir\") pod \"aws-ebs-csi-driver-node-fxzz5\" (UID: \"4f863b64-3884-435e-b469-06b94f479e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxzz5" Jul 29 04:03:41.462086 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.461645 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9384d535-332c-4320-a56e-a8cccbbbc5e8-konnectivity-ca\") pod \"konnectivity-agent-492cr\" (UID: \"9384d535-332c-4320-a56e-a8cccbbbc5e8\") " pod="kube-system/konnectivity-agent-492cr" Jul 29 04:03:41.462086 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.461669 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-run-systemd\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.463019 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.461694 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-run-openvswitch\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.463019 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.461727 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.463019 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.462087 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:03:41.463019 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:41.462157 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6vhn" podUID="350381e6-34a2-4f35-90d4-460ed24c7096" Jul 29 04:03:41.463644 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.463625 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-998k7\"" Jul 29 04:03:41.463850 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.463831 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 29 04:03:41.463954 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.463939 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 29 04:03:41.465793 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.465501 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.468536 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.467931 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.468536 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.468054 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 29 04:03:41.468810 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.468624 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 29 04:03:41.469193 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.469056 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zvr6g\"" Jul 29 04:03:41.470363 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.470168 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 29 04:03:41.470476 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.470419 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 29 04:03:41.470543 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.470516 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6887w\"" Jul 29 04:03:41.470714 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.470673 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:03:41.470915 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.470832 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 29 04:03:41.470915 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.470848 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 29 04:03:41.471107 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:41.470863 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9zpd" podUID="3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b" Jul 29 04:03:41.473342 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.473321 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v8d4n" Jul 29 04:03:41.475786 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.475764 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nc5xd\"" Jul 29 04:03:41.476217 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.475992 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 29 04:03:41.476217 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.476003 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 29 04:03:41.476217 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.476201 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2qc5b" Jul 29 04:03:41.478236 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.478215 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zjpnt" Jul 29 04:03:41.478805 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.478528 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 29 04:03:41.478908 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.478882 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 29 04:03:41.478988 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.478971 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 29 04:03:41.479046 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.478979 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5rtzq\"" Jul 29 04:03:41.480503 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.480485 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l9hbz\"" Jul 29 04:03:41.480816 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.480799 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 29 04:03:41.481006 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.480873 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 29 04:03:41.481878 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.481858 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7csrb" Jul 29 04:03:41.489357 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.489337 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7csrb" Jul 29 04:03:41.550352 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.550318 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 29 04:03:41.562501 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.562458 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0055fae1-bd7e-4e87-834c-3a317313ae0c-host\") pod \"node-ca-wwg6z\" (UID: \"0055fae1-bd7e-4e87-834c-3a317313ae0c\") " pod="openshift-image-registry/node-ca-wwg6z" Jul 29 04:03:41.562667 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.562523 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m8k8\" (UniqueName: \"kubernetes.io/projected/9fa6e244-be92-4245-a54e-7aa303802f4b-kube-api-access-8m8k8\") pod \"network-metrics-daemon-zbrk8\" (UID: \"9fa6e244-be92-4245-a54e-7aa303802f4b\") " pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:03:41.562667 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.562558 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc47e5bf-dc39-4b81-ba67-5c00203489e0-tmp-dir\") pod \"node-resolver-v8d4n\" (UID: \"dc47e5bf-dc39-4b81-ba67-5c00203489e0\") " pod="openshift-dns/node-resolver-v8d4n" Jul 29 04:03:41.562667 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.562586 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0055fae1-bd7e-4e87-834c-3a317313ae0c-host\") pod \"node-ca-wwg6z\" (UID: \"0055fae1-bd7e-4e87-834c-3a317313ae0c\") " pod="openshift-image-registry/node-ca-wwg6z" Jul 29 04:03:41.562667 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.562589 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddsgw\" (UniqueName: \"kubernetes.io/projected/e5d96939-0ee9-454b-a70b-1031f1c38623-kube-api-access-ddsgw\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.562667 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.562639 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-host-slash\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.562667 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.562664 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fa6e244-be92-4245-a54e-7aa303802f4b-metrics-certs\") pod \"network-metrics-daemon-zbrk8\" (UID: \"9fa6e244-be92-4245-a54e-7aa303802f4b\") " pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:03:41.562960 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.562693 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f863b64-3884-435e-b469-06b94f479e3b-socket-dir\") pod \"aws-ebs-csi-driver-node-fxzz5\" (UID: \"4f863b64-3884-435e-b469-06b94f479e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxzz5" Jul 29 04:03:41.562960 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.562721 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f863b64-3884-435e-b469-06b94f479e3b-registration-dir\") pod \"aws-ebs-csi-driver-node-fxzz5\" (UID: \"4f863b64-3884-435e-b469-06b94f479e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxzz5" Jul 29 04:03:41.562960 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.562752 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-etc-sysconfig\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.562960 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.562776 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-run\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.562960 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.562800 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4d18741-fe36-4f1a-a290-795bfc955fb0-cnibin\") pod \"multus-additional-cni-plugins-zjpnt\" (UID: \"a4d18741-fe36-4f1a-a290-795bfc955fb0\") " pod="openshift-multus/multus-additional-cni-plugins-zjpnt" Jul 29 04:03:41.562960 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.562824 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a4d18741-fe36-4f1a-a290-795bfc955fb0-os-release\") pod \"multus-additional-cni-plugins-zjpnt\" (UID: \"a4d18741-fe36-4f1a-a290-795bfc955fb0\") " pod="openshift-multus/multus-additional-cni-plugins-zjpnt" Jul 29 04:03:41.562960 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.562862 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f863b64-3884-435e-b469-06b94f479e3b-etc-selinux\") pod \"aws-ebs-csi-driver-node-fxzz5\" (UID: \"4f863b64-3884-435e-b469-06b94f479e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxzz5" Jul 29 04:03:41.562960 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.562889 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9384d535-332c-4320-a56e-a8cccbbbc5e8-agent-certs\") pod \"konnectivity-agent-492cr\" (UID: \"9384d535-332c-4320-a56e-a8cccbbbc5e8\") " pod="kube-system/konnectivity-agent-492cr" Jul 29 04:03:41.562960 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.562917 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gzps\" (UniqueName: \"kubernetes.io/projected/ed77db79-64ab-4dda-8b0c-3ab2ec321512-kube-api-access-7gzps\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.562960 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.562958 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-lib-modules\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.563472 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.563024 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-host-slash\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.563472 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.563087 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f863b64-3884-435e-b469-06b94f479e3b-etc-selinux\") pod \"aws-ebs-csi-driver-node-fxzz5\" (UID: \"4f863b64-3884-435e-b469-06b94f479e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxzz5" Jul 29 04:03:41.563472 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:41.563089 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 04:03:41.563472 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:41.563168 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fa6e244-be92-4245-a54e-7aa303802f4b-metrics-certs podName:9fa6e244-be92-4245-a54e-7aa303802f4b nodeName:}" failed. No retries permitted until 2026-07-29 04:03:42.063145771 +0000 UTC m=+2.922520505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9fa6e244-be92-4245-a54e-7aa303802f4b-metrics-certs") pod "network-metrics-daemon-zbrk8" (UID: "9fa6e244-be92-4245-a54e-7aa303802f4b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 04:03:41.563472 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.563218 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f863b64-3884-435e-b469-06b94f479e3b-socket-dir\") pod \"aws-ebs-csi-driver-node-fxzz5\" (UID: \"4f863b64-3884-435e-b469-06b94f479e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxzz5" Jul 29 04:03:41.563472 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.563268 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f863b64-3884-435e-b469-06b94f479e3b-registration-dir\") pod \"aws-ebs-csi-driver-node-fxzz5\" (UID: \"4f863b64-3884-435e-b469-06b94f479e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxzz5" Jul 29 04:03:41.563472 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.563322 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e5d96939-0ee9-454b-a70b-1031f1c38623-etc-tuned\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.563472 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.563431 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4d18741-fe36-4f1a-a290-795bfc955fb0-cni-binary-copy\") pod \"multus-additional-cni-plugins-zjpnt\" (UID: \"a4d18741-fe36-4f1a-a290-795bfc955fb0\") " pod="openshift-multus/multus-additional-cni-plugins-zjpnt" Jul 29 04:03:41.563904 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.563462 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-systemd-units\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.563904 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.563507 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-multus-cni-dir\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.563904 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.563531 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-cnibin\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.563904 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.563565 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b-kubelet-config\") pod \"global-pull-secret-syncer-b9zpd\" (UID: \"3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b\") " pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:03:41.563904 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.563591 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-run-openvswitch\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.563904 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.563619 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-etc-kubernetes\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.563904 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.563639 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-etc-modprobe-d\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.563904 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.563656 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5d96939-0ee9-454b-a70b-1031f1c38623-tmp\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.563904 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.563668 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 29 04:03:41.563904 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.563673 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp6zc\" (UniqueName: \"kubernetes.io/projected/0055fae1-bd7e-4e87-834c-3a317313ae0c-kube-api-access-dp6zc\") pod \"node-ca-wwg6z\" (UID: \"0055fae1-bd7e-4e87-834c-3a317313ae0c\") " pod="openshift-image-registry/node-ca-wwg6z" Jul 29 04:03:41.563904 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.563853 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2582x\" (UniqueName: \"kubernetes.io/projected/4f863b64-3884-435e-b469-06b94f479e3b-kube-api-access-2582x\") pod \"aws-ebs-csi-driver-node-fxzz5\" (UID: \"4f863b64-3884-435e-b469-06b94f479e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxzz5" Jul 29 04:03:41.563904 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.563860 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-systemd-units\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.563904 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.563881 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-etc-sysctl-conf\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.563904 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.563908 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-run-openvswitch\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.564641 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.563909 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-var-lib-kubelet\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.564641 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.563960 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a4d18741-fe36-4f1a-a290-795bfc955fb0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zjpnt\" (UID: \"a4d18741-fe36-4f1a-a290-795bfc955fb0\") " pod="openshift-multus/multus-additional-cni-plugins-zjpnt" Jul 29 04:03:41.564641 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.564038 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-log-socket\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.564641 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.564084 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7fab7167-d61f-486e-85d2-58004704568e-env-overrides\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.564641 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.564135 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-log-socket\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.564641 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.564199 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-system-cni-dir\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.564641 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.564296 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-host-run-k8s-cni-cncf-io\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.564641 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.564338 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-multus-conf-dir\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.564641 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.564471 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnt49\" (UniqueName: \"kubernetes.io/projected/dc47e5bf-dc39-4b81-ba67-5c00203489e0-kube-api-access-vnt49\") pod \"node-resolver-v8d4n\" (UID: \"dc47e5bf-dc39-4b81-ba67-5c00203489e0\") " pod="openshift-dns/node-resolver-v8d4n" Jul 29 04:03:41.564641 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.564514 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-host-kubelet\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.564641 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.564561 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-host-kubelet\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.564641 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.564592 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-host-run-netns\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.564641 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.564634 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-node-log\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.565271 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.564661 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-host-run-netns\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.565271 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.564678 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7fab7167-d61f-486e-85d2-58004704568e-ovn-node-metrics-cert\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.565271 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.564690 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7fab7167-d61f-486e-85d2-58004704568e-env-overrides\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.565271 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.564708 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-host-var-lib-cni-multus\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.565271 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.564734 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ed77db79-64ab-4dda-8b0c-3ab2ec321512-multus-daemon-config\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.565271 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.564750 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15922634-69e9-4620-ab51-f72a18853896-host-slash\") pod \"iptables-alerter-2qc5b\" (UID: \"15922634-69e9-4620-ab51-f72a18853896\") " pod="openshift-network-operator/iptables-alerter-2qc5b" Jul 29 04:03:41.565271 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.564697 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-node-log\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.565271 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.564772 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-674l9\" (UniqueName: \"kubernetes.io/projected/15922634-69e9-4620-ab51-f72a18853896-kube-api-access-674l9\") pod \"iptables-alerter-2qc5b\" (UID: \"15922634-69e9-4620-ab51-f72a18853896\") " pod="openshift-network-operator/iptables-alerter-2qc5b" Jul 29 04:03:41.565271 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.564798 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-host-cni-netd\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.565271 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.564819 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7fab7167-d61f-486e-85d2-58004704568e-ovnkube-config\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.565271 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.564856 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f863b64-3884-435e-b469-06b94f479e3b-sys-fs\") pod \"aws-ebs-csi-driver-node-fxzz5\" (UID: \"4f863b64-3884-435e-b469-06b94f479e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxzz5" Jul 29 04:03:41.565271 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.564864 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-host-cni-netd\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.565271 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.564881 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed77db79-64ab-4dda-8b0c-3ab2ec321512-cni-binary-copy\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.565271 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.564904 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-etc-sysctl-d\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.565271 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.564916 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f863b64-3884-435e-b469-06b94f479e3b-sys-fs\") pod \"aws-ebs-csi-driver-node-fxzz5\" (UID: \"4f863b64-3884-435e-b469-06b94f479e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxzz5" Jul 29 04:03:41.565271 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.564928 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-etc-systemd\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.565271 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.564950 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-sys\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.566144 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.564977 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-etc-openvswitch\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.566144 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565005 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f863b64-3884-435e-b469-06b94f479e3b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fxzz5\" (UID: \"4f863b64-3884-435e-b469-06b94f479e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxzz5" Jul 29 04:03:41.566144 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565028 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-os-release\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.566144 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565047 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-etc-openvswitch\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.566144 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565052 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-host-run-netns\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.566144 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565088 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b-original-pull-secret\") pod \"global-pull-secret-syncer-b9zpd\" (UID: \"3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b\") " pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:03:41.566144 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565115 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dc47e5bf-dc39-4b81-ba67-5c00203489e0-hosts-file\") pod \"node-resolver-v8d4n\" (UID: \"dc47e5bf-dc39-4b81-ba67-5c00203489e0\") " pod="openshift-dns/node-resolver-v8d4n" Jul 29 04:03:41.566144 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565124 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f863b64-3884-435e-b469-06b94f479e3b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fxzz5\" (UID: \"4f863b64-3884-435e-b469-06b94f479e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxzz5" Jul 29 04:03:41.566144 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565146 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-host-cni-bin\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.566144 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565177 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk987\" (UniqueName: \"kubernetes.io/projected/7fab7167-d61f-486e-85d2-58004704568e-kube-api-access-rk987\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.566144 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565245 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0055fae1-bd7e-4e87-834c-3a317313ae0c-serviceca\") pod \"node-ca-wwg6z\" (UID: \"0055fae1-bd7e-4e87-834c-3a317313ae0c\") " pod="openshift-image-registry/node-ca-wwg6z" Jul 29 04:03:41.566144 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565271 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-multus-socket-dir-parent\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.566144 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565276 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-host-cni-bin\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.566144 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565309 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-host-var-lib-cni-bin\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.566144 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565332 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-host\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.566144 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565350 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7fab7167-d61f-486e-85d2-58004704568e-ovnkube-config\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.566144 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565353 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a4d18741-fe36-4f1a-a290-795bfc955fb0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zjpnt\" (UID: \"a4d18741-fe36-4f1a-a290-795bfc955fb0\") " pod="openshift-multus/multus-additional-cni-plugins-zjpnt" Jul 29 04:03:41.566910 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565403 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f863b64-3884-435e-b469-06b94f479e3b-device-dir\") pod \"aws-ebs-csi-driver-node-fxzz5\" (UID: \"4f863b64-3884-435e-b469-06b94f479e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxzz5" Jul 29 04:03:41.566910 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565446 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9384d535-332c-4320-a56e-a8cccbbbc5e8-konnectivity-ca\") pod \"konnectivity-agent-492cr\" (UID: \"9384d535-332c-4320-a56e-a8cccbbbc5e8\") " pod="kube-system/konnectivity-agent-492cr" Jul 29 04:03:41.566910 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565475 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-etc-kubernetes\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.566910 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565485 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f863b64-3884-435e-b469-06b94f479e3b-device-dir\") pod \"aws-ebs-csi-driver-node-fxzz5\" (UID: \"4f863b64-3884-435e-b469-06b94f479e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxzz5" Jul 29 04:03:41.566910 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565517 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-run-systemd\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.566910 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565551 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.566910 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565581 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lqmq\" (UniqueName: \"kubernetes.io/projected/350381e6-34a2-4f35-90d4-460ed24c7096-kube-api-access-7lqmq\") pod \"network-check-target-x6vhn\" (UID: \"350381e6-34a2-4f35-90d4-460ed24c7096\") " pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:03:41.566910 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565590 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-run-systemd\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.566910 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565608 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b-dbus\") pod \"global-pull-secret-syncer-b9zpd\" (UID: \"3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b\") " pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:03:41.566910 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565636 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-var-lib-openvswitch\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.566910 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565663 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7fab7167-d61f-486e-85d2-58004704568e-ovnkube-script-lib\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.566910 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565689 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-host-var-lib-kubelet\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.566910 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565715 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-hostroot\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.566910 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565722 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0055fae1-bd7e-4e87-834c-3a317313ae0c-serviceca\") pod \"node-ca-wwg6z\" (UID: \"0055fae1-bd7e-4e87-834c-3a317313ae0c\") " pod="openshift-image-registry/node-ca-wwg6z" Jul 29 04:03:41.566910 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565739 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-host-run-multus-certs\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.566910 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565687 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.566910 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565766 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4d18741-fe36-4f1a-a290-795bfc955fb0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zjpnt\" (UID: \"a4d18741-fe36-4f1a-a290-795bfc955fb0\") " pod="openshift-multus/multus-additional-cni-plugins-zjpnt" Jul 29 04:03:41.568816 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565794 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9klr2\" (UniqueName: \"kubernetes.io/projected/a4d18741-fe36-4f1a-a290-795bfc955fb0-kube-api-access-9klr2\") pod \"multus-additional-cni-plugins-zjpnt\" (UID: \"a4d18741-fe36-4f1a-a290-795bfc955fb0\") " pod="openshift-multus/multus-additional-cni-plugins-zjpnt" Jul 29 04:03:41.568816 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565801 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-var-lib-openvswitch\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.568816 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565824 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-run-ovn\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.568816 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565859 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-host-run-ovn-kubernetes\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.568816 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565882 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-run-ovn\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.568816 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565888 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/15922634-69e9-4620-ab51-f72a18853896-iptables-alerter-script\") pod \"iptables-alerter-2qc5b\" (UID: \"15922634-69e9-4620-ab51-f72a18853896\") " pod="openshift-network-operator/iptables-alerter-2qc5b" Jul 29 04:03:41.568816 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565913 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4d18741-fe36-4f1a-a290-795bfc955fb0-system-cni-dir\") pod \"multus-additional-cni-plugins-zjpnt\" (UID: \"a4d18741-fe36-4f1a-a290-795bfc955fb0\") " pod="openshift-multus/multus-additional-cni-plugins-zjpnt" Jul 29 04:03:41.568816 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.565926 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fab7167-d61f-486e-85d2-58004704568e-host-run-ovn-kubernetes\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.568816 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.566066 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9384d535-332c-4320-a56e-a8cccbbbc5e8-konnectivity-ca\") pod \"konnectivity-agent-492cr\" (UID: \"9384d535-332c-4320-a56e-a8cccbbbc5e8\") " pod="kube-system/konnectivity-agent-492cr" Jul 29 04:03:41.568816 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.566239 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7fab7167-d61f-486e-85d2-58004704568e-ovnkube-script-lib\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.568816 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.568376 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9384d535-332c-4320-a56e-a8cccbbbc5e8-agent-certs\") pod \"konnectivity-agent-492cr\" (UID: \"9384d535-332c-4320-a56e-a8cccbbbc5e8\") " pod="kube-system/konnectivity-agent-492cr" Jul 29 04:03:41.568816 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.568444 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7fab7167-d61f-486e-85d2-58004704568e-ovn-node-metrics-cert\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.574211 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.574152 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m8k8\" (UniqueName: \"kubernetes.io/projected/9fa6e244-be92-4245-a54e-7aa303802f4b-kube-api-access-8m8k8\") pod \"network-metrics-daemon-zbrk8\" (UID: \"9fa6e244-be92-4245-a54e-7aa303802f4b\") " pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:03:41.575063 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.575039 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp6zc\" (UniqueName: \"kubernetes.io/projected/0055fae1-bd7e-4e87-834c-3a317313ae0c-kube-api-access-dp6zc\") pod \"node-ca-wwg6z\" (UID: \"0055fae1-bd7e-4e87-834c-3a317313ae0c\") " pod="openshift-image-registry/node-ca-wwg6z" Jul 29 04:03:41.576263 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.576237 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk987\" (UniqueName: \"kubernetes.io/projected/7fab7167-d61f-486e-85d2-58004704568e-kube-api-access-rk987\") pod \"ovnkube-node-p7792\" (UID: \"7fab7167-d61f-486e-85d2-58004704568e\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.576504 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.576482 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2582x\" (UniqueName: \"kubernetes.io/projected/4f863b64-3884-435e-b469-06b94f479e3b-kube-api-access-2582x\") pod \"aws-ebs-csi-driver-node-fxzz5\" (UID: \"4f863b64-3884-435e-b469-06b94f479e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxzz5" Jul 29 04:03:41.590103 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.590046 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-111.ec2.internal" event={"ID":"5aecb2ab7b558cf2544e36bba77ca9dc","Type":"ContainerStarted","Data":"35772bc6030192ec15480eaf98513516bb7e78b4936a3e265b71589e8197af3f"} Jul 29 04:03:41.592046 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.592019 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-111.ec2.internal" event={"ID":"e386db4858c90ca09ab480d4f37fcab1","Type":"ContainerStarted","Data":"bddcc0349211fd187f5f9fc86e7bc6e6c6254ef3501a8fbed6f5f224cf110148"} Jul 29 04:03:41.666609 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.666565 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gzps\" (UniqueName: \"kubernetes.io/projected/ed77db79-64ab-4dda-8b0c-3ab2ec321512-kube-api-access-7gzps\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.666819 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.666617 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-lib-modules\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.666819 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.666643 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e5d96939-0ee9-454b-a70b-1031f1c38623-etc-tuned\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.666819 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.666667 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4d18741-fe36-4f1a-a290-795bfc955fb0-cni-binary-copy\") pod \"multus-additional-cni-plugins-zjpnt\" (UID: \"a4d18741-fe36-4f1a-a290-795bfc955fb0\") " pod="openshift-multus/multus-additional-cni-plugins-zjpnt" Jul 29 04:03:41.666819 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.666689 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-multus-cni-dir\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.666819 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.666753 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-cnibin\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.666819 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.666798 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b-kubelet-config\") pod \"global-pull-secret-syncer-b9zpd\" (UID: \"3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b\") " pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:03:41.667146 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.666831 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-etc-kubernetes\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.667146 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.666856 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-etc-modprobe-d\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.667146 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.666881 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5d96939-0ee9-454b-a70b-1031f1c38623-tmp\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.667146 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.666911 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-etc-sysctl-conf\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.667146 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.666920 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b-kubelet-config\") pod \"global-pull-secret-syncer-b9zpd\" (UID: \"3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b\") " pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:03:41.667146 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.666935 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-var-lib-kubelet\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.667146 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.666941 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-multus-cni-dir\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.667146 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.666963 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a4d18741-fe36-4f1a-a290-795bfc955fb0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zjpnt\" (UID: \"a4d18741-fe36-4f1a-a290-795bfc955fb0\") " pod="openshift-multus/multus-additional-cni-plugins-zjpnt" Jul 29 04:03:41.667146 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.666982 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-cnibin\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.667146 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.666993 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-system-cni-dir\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.667146 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667020 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-host-run-k8s-cni-cncf-io\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.667146 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667032 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-var-lib-kubelet\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.667146 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667046 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-multus-conf-dir\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.667146 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667070 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnt49\" (UniqueName: \"kubernetes.io/projected/dc47e5bf-dc39-4b81-ba67-5c00203489e0-kube-api-access-vnt49\") pod \"node-resolver-v8d4n\" (UID: \"dc47e5bf-dc39-4b81-ba67-5c00203489e0\") " pod="openshift-dns/node-resolver-v8d4n" Jul 29 04:03:41.667146 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667099 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-host-var-lib-cni-multus\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.667146 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667120 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ed77db79-64ab-4dda-8b0c-3ab2ec321512-multus-daemon-config\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.667146 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667141 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15922634-69e9-4620-ab51-f72a18853896-host-slash\") pod \"iptables-alerter-2qc5b\" (UID: \"15922634-69e9-4620-ab51-f72a18853896\") " pod="openshift-network-operator/iptables-alerter-2qc5b" Jul 29 04:03:41.668006 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667164 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-etc-sysctl-conf\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.668006 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667170 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-674l9\" (UniqueName: \"kubernetes.io/projected/15922634-69e9-4620-ab51-f72a18853896-kube-api-access-674l9\") pod \"iptables-alerter-2qc5b\" (UID: \"15922634-69e9-4620-ab51-f72a18853896\") " pod="openshift-network-operator/iptables-alerter-2qc5b" Jul 29 04:03:41.668006 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667199 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed77db79-64ab-4dda-8b0c-3ab2ec321512-cni-binary-copy\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.668006 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667206 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-multus-conf-dir\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.668006 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667224 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-etc-sysctl-d\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.668006 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667269 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-etc-systemd\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.668006 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667290 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-sys\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.668006 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667323 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-lib-modules\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.668006 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667334 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-os-release\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.668006 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667363 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-host-run-k8s-cni-cncf-io\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.668006 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667378 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-host-run-netns\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.668006 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667408 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-system-cni-dir\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.668006 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667413 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b-original-pull-secret\") pod \"global-pull-secret-syncer-b9zpd\" (UID: \"3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b\") " pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:03:41.668006 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667405 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-host-var-lib-cni-multus\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.668006 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667453 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-etc-kubernetes\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.668006 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667456 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-os-release\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.668006 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667493 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-host-run-netns\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.668006 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667616 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-etc-modprobe-d\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.668994 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667686 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dc47e5bf-dc39-4b81-ba67-5c00203489e0-hosts-file\") pod \"node-resolver-v8d4n\" (UID: \"dc47e5bf-dc39-4b81-ba67-5c00203489e0\") " pod="openshift-dns/node-resolver-v8d4n" Jul 29 04:03:41.668994 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667714 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-etc-sysctl-d\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.668994 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667743 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-multus-socket-dir-parent\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.668994 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667774 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-etc-systemd\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.668994 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667773 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-host-var-lib-cni-bin\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.668994 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667814 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-host\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.668994 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667818 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-host-var-lib-cni-bin\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.668994 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667847 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a4d18741-fe36-4f1a-a290-795bfc955fb0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zjpnt\" (UID: \"a4d18741-fe36-4f1a-a290-795bfc955fb0\") " pod="openshift-multus/multus-additional-cni-plugins-zjpnt" Jul 29 04:03:41.668994 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667870 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dc47e5bf-dc39-4b81-ba67-5c00203489e0-hosts-file\") pod \"node-resolver-v8d4n\" (UID: \"dc47e5bf-dc39-4b81-ba67-5c00203489e0\") " pod="openshift-dns/node-resolver-v8d4n" Jul 29 04:03:41.668994 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667879 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-etc-kubernetes\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.668994 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667910 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lqmq\" (UniqueName: \"kubernetes.io/projected/350381e6-34a2-4f35-90d4-460ed24c7096-kube-api-access-7lqmq\") pod \"network-check-target-x6vhn\" (UID: \"350381e6-34a2-4f35-90d4-460ed24c7096\") " pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:03:41.668994 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667923 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-multus-socket-dir-parent\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.668994 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667937 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b-dbus\") pod \"global-pull-secret-syncer-b9zpd\" (UID: \"3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b\") " pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:03:41.668994 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667986 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-sys\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.668994 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667987 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-host-var-lib-kubelet\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.668994 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.667995 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a4d18741-fe36-4f1a-a290-795bfc955fb0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zjpnt\" (UID: \"a4d18741-fe36-4f1a-a290-795bfc955fb0\") " pod="openshift-multus/multus-additional-cni-plugins-zjpnt" Jul 29 04:03:41.668994 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.668022 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-hostroot\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.668994 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.668023 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-host-var-lib-kubelet\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.669927 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.668055 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ed77db79-64ab-4dda-8b0c-3ab2ec321512-multus-daemon-config\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.669927 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.668060 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed77db79-64ab-4dda-8b0c-3ab2ec321512-cni-binary-copy\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.669927 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.668055 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-host-run-multus-certs\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.669927 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:41.668090 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 04:03:41.669927 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.668111 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4d18741-fe36-4f1a-a290-795bfc955fb0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zjpnt\" (UID: \"a4d18741-fe36-4f1a-a290-795bfc955fb0\") " pod="openshift-multus/multus-additional-cni-plugins-zjpnt" Jul 29 04:03:41.669927 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.668092 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-host\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.669927 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.668140 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b-dbus\") pod \"global-pull-secret-syncer-b9zpd\" (UID: \"3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b\") " pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:03:41.669927 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:41.668147 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b-original-pull-secret podName:3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b nodeName:}" failed. No retries permitted until 2026-07-29 04:03:42.168128038 +0000 UTC m=+3.027502755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b-original-pull-secret") pod "global-pull-secret-syncer-b9zpd" (UID: "3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b") : object "kube-system"/"original-pull-secret" not registered Jul 29 04:03:41.669927 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.668089 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-hostroot\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.669927 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.668170 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9klr2\" (UniqueName: \"kubernetes.io/projected/a4d18741-fe36-4f1a-a290-795bfc955fb0-kube-api-access-9klr2\") pod \"multus-additional-cni-plugins-zjpnt\" (UID: \"a4d18741-fe36-4f1a-a290-795bfc955fb0\") " pod="openshift-multus/multus-additional-cni-plugins-zjpnt" Jul 29 04:03:41.669927 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.668201 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/15922634-69e9-4620-ab51-f72a18853896-iptables-alerter-script\") pod \"iptables-alerter-2qc5b\" (UID: \"15922634-69e9-4620-ab51-f72a18853896\") " pod="openshift-network-operator/iptables-alerter-2qc5b" Jul 29 04:03:41.669927 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.668232 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4d18741-fe36-4f1a-a290-795bfc955fb0-system-cni-dir\") pod \"multus-additional-cni-plugins-zjpnt\" (UID: \"a4d18741-fe36-4f1a-a290-795bfc955fb0\") " pod="openshift-multus/multus-additional-cni-plugins-zjpnt" Jul 29 04:03:41.669927 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.668257 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed77db79-64ab-4dda-8b0c-3ab2ec321512-host-run-multus-certs\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.669927 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.668295 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc47e5bf-dc39-4b81-ba67-5c00203489e0-tmp-dir\") pod \"node-resolver-v8d4n\" (UID: \"dc47e5bf-dc39-4b81-ba67-5c00203489e0\") " pod="openshift-dns/node-resolver-v8d4n" Jul 29 04:03:41.669927 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.668317 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4d18741-fe36-4f1a-a290-795bfc955fb0-system-cni-dir\") pod \"multus-additional-cni-plugins-zjpnt\" (UID: \"a4d18741-fe36-4f1a-a290-795bfc955fb0\") " pod="openshift-multus/multus-additional-cni-plugins-zjpnt" Jul 29 04:03:41.669927 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.668335 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddsgw\" (UniqueName: \"kubernetes.io/projected/e5d96939-0ee9-454b-a70b-1031f1c38623-kube-api-access-ddsgw\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.669927 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.668371 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-etc-sysconfig\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.670865 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.668400 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-run\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.670865 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.668427 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4d18741-fe36-4f1a-a290-795bfc955fb0-cnibin\") pod \"multus-additional-cni-plugins-zjpnt\" (UID: \"a4d18741-fe36-4f1a-a290-795bfc955fb0\") " pod="openshift-multus/multus-additional-cni-plugins-zjpnt" Jul 29 04:03:41.670865 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.668473 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a4d18741-fe36-4f1a-a290-795bfc955fb0-os-release\") pod \"multus-additional-cni-plugins-zjpnt\" (UID: \"a4d18741-fe36-4f1a-a290-795bfc955fb0\") " pod="openshift-multus/multus-additional-cni-plugins-zjpnt" Jul 29 04:03:41.670865 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.668579 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a4d18741-fe36-4f1a-a290-795bfc955fb0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zjpnt\" (UID: \"a4d18741-fe36-4f1a-a290-795bfc955fb0\") " pod="openshift-multus/multus-additional-cni-plugins-zjpnt" Jul 29 04:03:41.670865 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.668647 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a4d18741-fe36-4f1a-a290-795bfc955fb0-os-release\") pod \"multus-additional-cni-plugins-zjpnt\" (UID: \"a4d18741-fe36-4f1a-a290-795bfc955fb0\") " pod="openshift-multus/multus-additional-cni-plugins-zjpnt" Jul 29 04:03:41.670865 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.668705 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-etc-sysconfig\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.670865 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.668775 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-run\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.670865 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.668816 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4d18741-fe36-4f1a-a290-795bfc955fb0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zjpnt\" (UID: \"a4d18741-fe36-4f1a-a290-795bfc955fb0\") " pod="openshift-multus/multus-additional-cni-plugins-zjpnt" Jul 29 04:03:41.670865 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.668831 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4d18741-fe36-4f1a-a290-795bfc955fb0-cnibin\") pod \"multus-additional-cni-plugins-zjpnt\" (UID: \"a4d18741-fe36-4f1a-a290-795bfc955fb0\") " pod="openshift-multus/multus-additional-cni-plugins-zjpnt" Jul 29 04:03:41.670865 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.668850 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/15922634-69e9-4620-ab51-f72a18853896-iptables-alerter-script\") pod \"iptables-alerter-2qc5b\" (UID: \"15922634-69e9-4620-ab51-f72a18853896\") " pod="openshift-network-operator/iptables-alerter-2qc5b" Jul 29 04:03:41.670865 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.668861 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5d96939-0ee9-454b-a70b-1031f1c38623-etc-kubernetes\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.670865 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.668870 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15922634-69e9-4620-ab51-f72a18853896-host-slash\") pod \"iptables-alerter-2qc5b\" (UID: \"15922634-69e9-4620-ab51-f72a18853896\") " pod="openshift-network-operator/iptables-alerter-2qc5b" Jul 29 04:03:41.670865 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.669016 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4d18741-fe36-4f1a-a290-795bfc955fb0-cni-binary-copy\") pod \"multus-additional-cni-plugins-zjpnt\" (UID: \"a4d18741-fe36-4f1a-a290-795bfc955fb0\") " pod="openshift-multus/multus-additional-cni-plugins-zjpnt" Jul 29 04:03:41.670865 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.669130 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc47e5bf-dc39-4b81-ba67-5c00203489e0-tmp-dir\") pod \"node-resolver-v8d4n\" (UID: \"dc47e5bf-dc39-4b81-ba67-5c00203489e0\") " pod="openshift-dns/node-resolver-v8d4n" Jul 29 04:03:41.671401 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.670955 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5d96939-0ee9-454b-a70b-1031f1c38623-tmp\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.671401 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.671066 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e5d96939-0ee9-454b-a70b-1031f1c38623-etc-tuned\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.676420 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:41.676386 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 04:03:41.676420 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:41.676422 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 04:03:41.676699 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:41.676441 2579 projected.go:196] Error preparing data for projected volume kube-api-access-7lqmq for pod openshift-network-diagnostics/network-check-target-x6vhn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 04:03:41.676699 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:41.676539 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/350381e6-34a2-4f35-90d4-460ed24c7096-kube-api-access-7lqmq podName:350381e6-34a2-4f35-90d4-460ed24c7096 nodeName:}" failed. No retries permitted until 2026-07-29 04:03:42.176518626 +0000 UTC m=+3.035893332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7lqmq" (UniqueName: "kubernetes.io/projected/350381e6-34a2-4f35-90d4-460ed24c7096-kube-api-access-7lqmq") pod "network-check-target-x6vhn" (UID: "350381e6-34a2-4f35-90d4-460ed24c7096") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 04:03:41.678478 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.678412 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gzps\" (UniqueName: \"kubernetes.io/projected/ed77db79-64ab-4dda-8b0c-3ab2ec321512-kube-api-access-7gzps\") pod \"multus-snqz4\" (UID: \"ed77db79-64ab-4dda-8b0c-3ab2ec321512\") " pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.678622 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.678601 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnt49\" (UniqueName: \"kubernetes.io/projected/dc47e5bf-dc39-4b81-ba67-5c00203489e0-kube-api-access-vnt49\") pod \"node-resolver-v8d4n\" (UID: \"dc47e5bf-dc39-4b81-ba67-5c00203489e0\") " pod="openshift-dns/node-resolver-v8d4n" Jul 29 04:03:41.678824 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.678803 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-674l9\" (UniqueName: \"kubernetes.io/projected/15922634-69e9-4620-ab51-f72a18853896-kube-api-access-674l9\") pod \"iptables-alerter-2qc5b\" (UID: \"15922634-69e9-4620-ab51-f72a18853896\") " pod="openshift-network-operator/iptables-alerter-2qc5b" Jul 29 04:03:41.679988 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.679963 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddsgw\" (UniqueName: \"kubernetes.io/projected/e5d96939-0ee9-454b-a70b-1031f1c38623-kube-api-access-ddsgw\") pod \"tuned-tf8wh\" (UID: \"e5d96939-0ee9-454b-a70b-1031f1c38623\") " pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.680235 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.680209 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9klr2\" (UniqueName: \"kubernetes.io/projected/a4d18741-fe36-4f1a-a290-795bfc955fb0-kube-api-access-9klr2\") pod \"multus-additional-cni-plugins-zjpnt\" (UID: \"a4d18741-fe36-4f1a-a290-795bfc955fb0\") " pod="openshift-multus/multus-additional-cni-plugins-zjpnt" Jul 29 04:03:41.760586 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.760539 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:03:41.773193 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.773164 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wwg6z" Jul 29 04:03:41.781346 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.781318 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxzz5" Jul 29 04:03:41.802309 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.802279 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-492cr" Jul 29 04:03:41.810370 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.810348 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" Jul 29 04:03:41.818406 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.818383 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-snqz4" Jul 29 04:03:41.821635 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.821614 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 04:03:41.826408 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.826381 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v8d4n" Jul 29 04:03:41.834358 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.834333 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2qc5b" Jul 29 04:03:41.841332 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:41.841310 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zjpnt" Jul 29 04:03:42.071505 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:42.071410 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fa6e244-be92-4245-a54e-7aa303802f4b-metrics-certs\") pod \"network-metrics-daemon-zbrk8\" (UID: \"9fa6e244-be92-4245-a54e-7aa303802f4b\") " pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:03:42.071675 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:42.071551 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 04:03:42.071675 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:42.071614 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fa6e244-be92-4245-a54e-7aa303802f4b-metrics-certs podName:9fa6e244-be92-4245-a54e-7aa303802f4b nodeName:}" failed. No retries permitted until 2026-07-29 04:03:43.071599437 +0000 UTC m=+3.930974156 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9fa6e244-be92-4245-a54e-7aa303802f4b-metrics-certs") pod "network-metrics-daemon-zbrk8" (UID: "9fa6e244-be92-4245-a54e-7aa303802f4b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 04:03:42.172635 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:42.172588 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b-original-pull-secret\") pod \"global-pull-secret-syncer-b9zpd\" (UID: \"3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b\") " pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:03:42.172786 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:42.172766 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 04:03:42.172855 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:42.172841 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b-original-pull-secret podName:3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b nodeName:}" failed. No retries permitted until 2026-07-29 04:03:43.172822135 +0000 UTC m=+4.032196841 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b-original-pull-secret") pod "global-pull-secret-syncer-b9zpd" (UID: "3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b") : object "kube-system"/"original-pull-secret" not registered Jul 29 04:03:42.273920 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:42.273873 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lqmq\" (UniqueName: \"kubernetes.io/projected/350381e6-34a2-4f35-90d4-460ed24c7096-kube-api-access-7lqmq\") pod \"network-check-target-x6vhn\" (UID: \"350381e6-34a2-4f35-90d4-460ed24c7096\") " pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:03:42.274097 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:42.274072 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 04:03:42.274143 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:42.274105 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 04:03:42.274143 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:42.274122 2579 projected.go:196] Error preparing data for projected volume kube-api-access-7lqmq for pod openshift-network-diagnostics/network-check-target-x6vhn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 04:03:42.274208 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:42.274199 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/350381e6-34a2-4f35-90d4-460ed24c7096-kube-api-access-7lqmq podName:350381e6-34a2-4f35-90d4-460ed24c7096 nodeName:}" failed. No retries permitted until 2026-07-29 04:03:43.274176263 +0000 UTC m=+4.133550990 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lqmq" (UniqueName: "kubernetes.io/projected/350381e6-34a2-4f35-90d4-460ed24c7096-kube-api-access-7lqmq") pod "network-check-target-x6vhn" (UID: "350381e6-34a2-4f35-90d4-460ed24c7096") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 04:03:42.495354 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:42.495288 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 03:58:41 +0000 UTC" deadline="2028-04-22 20:40:34.892415454 +0000 UTC" Jul 29 04:03:42.495354 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:42.495348 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15208h36m52.397071001s" Jul 29 04:03:42.585760 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:42.585732 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:03:42.585919 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:42.585883 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbrk8" podUID="9fa6e244-be92-4245-a54e-7aa303802f4b" Jul 29 04:03:42.599672 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:42.599630 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded77db79_64ab_4dda_8b0c_3ab2ec321512.slice/crio-0f5d90b65f54333e4550d48d71931c4e8c4b788e6c379808c524d87163222962 WatchSource:0}: Error finding container 0f5d90b65f54333e4550d48d71931c4e8c4b788e6c379808c524d87163222962: Status 404 returned error can't find the container with id 0f5d90b65f54333e4550d48d71931c4e8c4b788e6c379808c524d87163222962 Jul 29 04:03:42.601601 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:42.601572 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15922634_69e9_4620_ab51_f72a18853896.slice/crio-b2feb545c77c2dfe6e9215345bd77e4ed0bd8ca3a745a27d2c8f23e134316bd2 WatchSource:0}: Error finding container b2feb545c77c2dfe6e9215345bd77e4ed0bd8ca3a745a27d2c8f23e134316bd2: Status 404 returned error can't find the container with id b2feb545c77c2dfe6e9215345bd77e4ed0bd8ca3a745a27d2c8f23e134316bd2 Jul 29 04:03:42.602447 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:42.602338 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fab7167_d61f_486e_85d2_58004704568e.slice/crio-0735bb9902cd5919c965f09fab3dd85564bdb589edf3e982760d97f94782dc17 WatchSource:0}: Error finding container 0735bb9902cd5919c965f09fab3dd85564bdb589edf3e982760d97f94782dc17: Status 404 returned error can't find the container with id 0735bb9902cd5919c965f09fab3dd85564bdb589edf3e982760d97f94782dc17 Jul 29 04:03:42.606711 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:42.606689 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0055fae1_bd7e_4e87_834c_3a317313ae0c.slice/crio-e4c5e8acabe5966dcd3574fccb0280bf07792b9a34d4f4c7797d8637455f70ce WatchSource:0}: Error finding container e4c5e8acabe5966dcd3574fccb0280bf07792b9a34d4f4c7797d8637455f70ce: Status 404 returned error can't find the container with id e4c5e8acabe5966dcd3574fccb0280bf07792b9a34d4f4c7797d8637455f70ce Jul 29 04:03:42.630170 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:42.630069 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc47e5bf_dc39_4b81_ba67_5c00203489e0.slice/crio-adf82a6a372eceb1b0de3074f5087f9501a872de9f13b41a6c8200826e7255ca WatchSource:0}: Error finding container adf82a6a372eceb1b0de3074f5087f9501a872de9f13b41a6c8200826e7255ca: Status 404 returned error can't find the container with id adf82a6a372eceb1b0de3074f5087f9501a872de9f13b41a6c8200826e7255ca Jul 29 04:03:42.630982 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:42.630957 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d18741_fe36_4f1a_a290_795bfc955fb0.slice/crio-20b403fe41213d87e74eadc236d4025a7e3452642582dfadec40d46aff972c08 WatchSource:0}: Error finding container 20b403fe41213d87e74eadc236d4025a7e3452642582dfadec40d46aff972c08: Status 404 returned error can't find the container with id 20b403fe41213d87e74eadc236d4025a7e3452642582dfadec40d46aff972c08 Jul 29 04:03:42.631884 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:42.631812 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f863b64_3884_435e_b469_06b94f479e3b.slice/crio-693dc5adae8db90cfd46fa6e22d2392786735ef52018fddb3bfc0a97e740e475 WatchSource:0}: Error finding container 693dc5adae8db90cfd46fa6e22d2392786735ef52018fddb3bfc0a97e740e475: Status 404 returned error can't find the container with id 693dc5adae8db90cfd46fa6e22d2392786735ef52018fddb3bfc0a97e740e475 Jul 29 04:03:42.633177 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:03:42.633156 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9384d535_332c_4320_a56e_a8cccbbbc5e8.slice/crio-85be85d16f31e2079a2778c29691e68d05b3f11d2f8d881bb92f3043e1e490c8 WatchSource:0}: Error finding container 85be85d16f31e2079a2778c29691e68d05b3f11d2f8d881bb92f3043e1e490c8: Status 404 returned error can't find the container with id 85be85d16f31e2079a2778c29691e68d05b3f11d2f8d881bb92f3043e1e490c8 Jul 29 04:03:43.079723 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:43.079414 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fa6e244-be92-4245-a54e-7aa303802f4b-metrics-certs\") pod \"network-metrics-daemon-zbrk8\" (UID: \"9fa6e244-be92-4245-a54e-7aa303802f4b\") " pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:03:43.079889 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:43.079767 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 04:03:43.079889 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:43.079849 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fa6e244-be92-4245-a54e-7aa303802f4b-metrics-certs podName:9fa6e244-be92-4245-a54e-7aa303802f4b nodeName:}" failed. No retries permitted until 2026-07-29 04:03:45.079827524 +0000 UTC m=+5.939202246 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9fa6e244-be92-4245-a54e-7aa303802f4b-metrics-certs") pod "network-metrics-daemon-zbrk8" (UID: "9fa6e244-be92-4245-a54e-7aa303802f4b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 04:03:43.180733 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:43.180691 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b-original-pull-secret\") pod \"global-pull-secret-syncer-b9zpd\" (UID: \"3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b\") " pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:03:43.180910 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:43.180872 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 04:03:43.180971 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:43.180933 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b-original-pull-secret podName:3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b nodeName:}" failed. No retries permitted until 2026-07-29 04:03:45.180914528 +0000 UTC m=+6.040289244 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b-original-pull-secret") pod "global-pull-secret-syncer-b9zpd" (UID: "3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b") : object "kube-system"/"original-pull-secret" not registered Jul 29 04:03:43.281880 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:43.281838 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lqmq\" (UniqueName: \"kubernetes.io/projected/350381e6-34a2-4f35-90d4-460ed24c7096-kube-api-access-7lqmq\") pod \"network-check-target-x6vhn\" (UID: \"350381e6-34a2-4f35-90d4-460ed24c7096\") " pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:03:43.282096 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:43.282078 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 04:03:43.282201 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:43.282105 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 04:03:43.282201 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:43.282126 2579 projected.go:196] Error preparing data for projected volume kube-api-access-7lqmq for pod openshift-network-diagnostics/network-check-target-x6vhn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 04:03:43.282201 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:43.282184 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/350381e6-34a2-4f35-90d4-460ed24c7096-kube-api-access-7lqmq podName:350381e6-34a2-4f35-90d4-460ed24c7096 nodeName:}" failed. No retries permitted until 2026-07-29 04:03:45.28216654 +0000 UTC m=+6.141541263 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lqmq" (UniqueName: "kubernetes.io/projected/350381e6-34a2-4f35-90d4-460ed24c7096-kube-api-access-7lqmq") pod "network-check-target-x6vhn" (UID: "350381e6-34a2-4f35-90d4-460ed24c7096") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 04:03:43.496232 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:43.496173 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 03:58:41 +0000 UTC" deadline="2028-01-26 23:29:51.406816478 +0000 UTC" Jul 29 04:03:43.496232 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:43.496225 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13123h26m7.910596367s" Jul 29 04:03:43.588943 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:43.588170 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:03:43.588943 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:43.588310 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6vhn" podUID="350381e6-34a2-4f35-90d4-460ed24c7096" Jul 29 04:03:43.588943 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:43.588728 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:03:43.588943 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:43.588826 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9zpd" podUID="3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b" Jul 29 04:03:43.603016 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:43.602982 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-492cr" event={"ID":"9384d535-332c-4320-a56e-a8cccbbbc5e8","Type":"ContainerStarted","Data":"85be85d16f31e2079a2778c29691e68d05b3f11d2f8d881bb92f3043e1e490c8"} Jul 29 04:03:43.608778 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:43.608572 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxzz5" event={"ID":"4f863b64-3884-435e-b469-06b94f479e3b","Type":"ContainerStarted","Data":"693dc5adae8db90cfd46fa6e22d2392786735ef52018fddb3bfc0a97e740e475"} Jul 29 04:03:43.621279 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:43.621234 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" event={"ID":"e5d96939-0ee9-454b-a70b-1031f1c38623","Type":"ContainerStarted","Data":"3938219ebc30dd114d532b8bd6234bdde7a7b4a09eab0ca1eb4e237ff8c7c517"} Jul 29 04:03:43.628936 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:43.628837 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v8d4n" event={"ID":"dc47e5bf-dc39-4b81-ba67-5c00203489e0","Type":"ContainerStarted","Data":"adf82a6a372eceb1b0de3074f5087f9501a872de9f13b41a6c8200826e7255ca"} Jul 29 04:03:43.639317 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:43.639264 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wwg6z" event={"ID":"0055fae1-bd7e-4e87-834c-3a317313ae0c","Type":"ContainerStarted","Data":"e4c5e8acabe5966dcd3574fccb0280bf07792b9a34d4f4c7797d8637455f70ce"} Jul 29 04:03:43.646995 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:43.646959 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7792" event={"ID":"7fab7167-d61f-486e-85d2-58004704568e","Type":"ContainerStarted","Data":"0735bb9902cd5919c965f09fab3dd85564bdb589edf3e982760d97f94782dc17"} Jul 29 04:03:43.653404 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:43.653373 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2qc5b" event={"ID":"15922634-69e9-4620-ab51-f72a18853896","Type":"ContainerStarted","Data":"b2feb545c77c2dfe6e9215345bd77e4ed0bd8ca3a745a27d2c8f23e134316bd2"} Jul 29 04:03:43.659371 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:43.659295 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-snqz4" event={"ID":"ed77db79-64ab-4dda-8b0c-3ab2ec321512","Type":"ContainerStarted","Data":"0f5d90b65f54333e4550d48d71931c4e8c4b788e6c379808c524d87163222962"} Jul 29 04:03:43.671865 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:43.671687 2579 generic.go:356] "Generic (PLEG): container finished" podID="5aecb2ab7b558cf2544e36bba77ca9dc" containerID="85617f70712e982bbacc917dde08eecdc4a6922d5477874a7575f08720f2206b" exitCode=0 Jul 29 04:03:43.671865 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:43.671778 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-111.ec2.internal" event={"ID":"5aecb2ab7b558cf2544e36bba77ca9dc","Type":"ContainerDied","Data":"85617f70712e982bbacc917dde08eecdc4a6922d5477874a7575f08720f2206b"} Jul 29 04:03:43.693748 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:43.693672 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-111.ec2.internal" event={"ID":"e386db4858c90ca09ab480d4f37fcab1","Type":"ContainerStarted","Data":"241532662ccec311df57c994383b03e9567976365c7b4b77a71c592c2b263428"} Jul 29 04:03:43.703944 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:43.703909 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjpnt" event={"ID":"a4d18741-fe36-4f1a-a290-795bfc955fb0","Type":"ContainerStarted","Data":"20b403fe41213d87e74eadc236d4025a7e3452642582dfadec40d46aff972c08"} Jul 29 04:03:44.585743 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:44.585714 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:03:44.586223 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:44.585855 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbrk8" podUID="9fa6e244-be92-4245-a54e-7aa303802f4b" Jul 29 04:03:44.721386 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:44.721348 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-111.ec2.internal" event={"ID":"5aecb2ab7b558cf2544e36bba77ca9dc","Type":"ContainerStarted","Data":"0c64a0f9c8fe2d664dc14f26abd6367383dba62e5dea4bc2891ff72a74aea409"} Jul 29 04:03:44.736778 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:44.736719 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-111.ec2.internal" podStartSLOduration=4.736698476 podStartE2EDuration="4.736698476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 04:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:03:43.708231125 +0000 UTC m=+4.567605847" watchObservedRunningTime="2026-07-29 04:03:44.736698476 +0000 UTC m=+5.596073202" Jul 29 04:03:44.737000 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:44.736964 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-111.ec2.internal" podStartSLOduration=4.736951338 podStartE2EDuration="4.736951338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 04:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:03:44.735752134 +0000 UTC m=+5.595126859" watchObservedRunningTime="2026-07-29 04:03:44.736951338 +0000 UTC m=+5.596326065" Jul 29 04:03:45.100334 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:45.100295 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fa6e244-be92-4245-a54e-7aa303802f4b-metrics-certs\") pod \"network-metrics-daemon-zbrk8\" (UID: \"9fa6e244-be92-4245-a54e-7aa303802f4b\") " pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:03:45.100535 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:45.100517 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 04:03:45.100601 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:45.100590 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fa6e244-be92-4245-a54e-7aa303802f4b-metrics-certs podName:9fa6e244-be92-4245-a54e-7aa303802f4b nodeName:}" failed. No retries permitted until 2026-07-29 04:03:49.100571633 +0000 UTC m=+9.959946349 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9fa6e244-be92-4245-a54e-7aa303802f4b-metrics-certs") pod "network-metrics-daemon-zbrk8" (UID: "9fa6e244-be92-4245-a54e-7aa303802f4b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 04:03:45.201591 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:45.201553 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b-original-pull-secret\") pod \"global-pull-secret-syncer-b9zpd\" (UID: \"3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b\") " pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:03:45.201753 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:45.201732 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 04:03:45.201823 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:45.201806 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b-original-pull-secret podName:3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b nodeName:}" failed. No retries permitted until 2026-07-29 04:03:49.201786905 +0000 UTC m=+10.061161621 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b-original-pull-secret") pod "global-pull-secret-syncer-b9zpd" (UID: "3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b") : object "kube-system"/"original-pull-secret" not registered Jul 29 04:03:45.302224 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:45.302176 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lqmq\" (UniqueName: \"kubernetes.io/projected/350381e6-34a2-4f35-90d4-460ed24c7096-kube-api-access-7lqmq\") pod \"network-check-target-x6vhn\" (UID: \"350381e6-34a2-4f35-90d4-460ed24c7096\") " pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:03:45.302415 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:45.302341 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 04:03:45.302415 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:45.302366 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 04:03:45.302415 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:45.302380 2579 projected.go:196] Error preparing data for projected volume kube-api-access-7lqmq for pod openshift-network-diagnostics/network-check-target-x6vhn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 04:03:45.302572 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:45.302444 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/350381e6-34a2-4f35-90d4-460ed24c7096-kube-api-access-7lqmq podName:350381e6-34a2-4f35-90d4-460ed24c7096 nodeName:}" failed. No retries permitted until 2026-07-29 04:03:49.302425318 +0000 UTC m=+10.161800043 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lqmq" (UniqueName: "kubernetes.io/projected/350381e6-34a2-4f35-90d4-460ed24c7096-kube-api-access-7lqmq") pod "network-check-target-x6vhn" (UID: "350381e6-34a2-4f35-90d4-460ed24c7096") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 04:03:45.588313 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:45.588282 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:03:45.588774 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:45.588403 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6vhn" podUID="350381e6-34a2-4f35-90d4-460ed24c7096" Jul 29 04:03:45.588840 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:45.588768 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:03:45.588890 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:45.588864 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9zpd" podUID="3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b" Jul 29 04:03:46.586152 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:46.586108 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:03:46.586355 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:46.586284 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbrk8" podUID="9fa6e244-be92-4245-a54e-7aa303802f4b" Jul 29 04:03:47.586610 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:47.586517 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:03:47.587081 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:47.586652 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6vhn" podUID="350381e6-34a2-4f35-90d4-460ed24c7096" Jul 29 04:03:47.587982 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:47.587958 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:03:47.588123 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:47.588097 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9zpd" podUID="3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b" Jul 29 04:03:48.586138 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:48.586102 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:03:48.586335 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:48.586265 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbrk8" podUID="9fa6e244-be92-4245-a54e-7aa303802f4b" Jul 29 04:03:49.133090 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:49.133049 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fa6e244-be92-4245-a54e-7aa303802f4b-metrics-certs\") pod \"network-metrics-daemon-zbrk8\" (UID: \"9fa6e244-be92-4245-a54e-7aa303802f4b\") " pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:03:49.133580 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:49.133289 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 04:03:49.133580 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:49.133360 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fa6e244-be92-4245-a54e-7aa303802f4b-metrics-certs podName:9fa6e244-be92-4245-a54e-7aa303802f4b nodeName:}" failed. No retries permitted until 2026-07-29 04:03:57.133340101 +0000 UTC m=+17.992714816 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9fa6e244-be92-4245-a54e-7aa303802f4b-metrics-certs") pod "network-metrics-daemon-zbrk8" (UID: "9fa6e244-be92-4245-a54e-7aa303802f4b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 04:03:49.234565 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:49.234493 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b-original-pull-secret\") pod \"global-pull-secret-syncer-b9zpd\" (UID: \"3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b\") " pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:03:49.234728 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:49.234662 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 04:03:49.234803 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:49.234735 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b-original-pull-secret podName:3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b nodeName:}" failed. No retries permitted until 2026-07-29 04:03:57.234713861 +0000 UTC m=+18.094088583 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b-original-pull-secret") pod "global-pull-secret-syncer-b9zpd" (UID: "3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b") : object "kube-system"/"original-pull-secret" not registered Jul 29 04:03:49.335821 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:49.335780 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lqmq\" (UniqueName: \"kubernetes.io/projected/350381e6-34a2-4f35-90d4-460ed24c7096-kube-api-access-7lqmq\") pod \"network-check-target-x6vhn\" (UID: \"350381e6-34a2-4f35-90d4-460ed24c7096\") " pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:03:49.336006 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:49.335964 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 04:03:49.336006 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:49.335993 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 04:03:49.336006 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:49.336006 2579 projected.go:196] Error preparing data for projected volume kube-api-access-7lqmq for pod openshift-network-diagnostics/network-check-target-x6vhn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 04:03:49.336170 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:49.336069 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/350381e6-34a2-4f35-90d4-460ed24c7096-kube-api-access-7lqmq podName:350381e6-34a2-4f35-90d4-460ed24c7096 nodeName:}" failed. No retries permitted until 2026-07-29 04:03:57.336050446 +0000 UTC m=+18.195425151 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lqmq" (UniqueName: "kubernetes.io/projected/350381e6-34a2-4f35-90d4-460ed24c7096-kube-api-access-7lqmq") pod "network-check-target-x6vhn" (UID: "350381e6-34a2-4f35-90d4-460ed24c7096") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 04:03:49.587025 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:49.586688 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:03:49.587025 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:49.586852 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6vhn" podUID="350381e6-34a2-4f35-90d4-460ed24c7096" Jul 29 04:03:49.587296 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:49.587278 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:03:49.587429 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:49.587384 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9zpd" podUID="3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b" Jul 29 04:03:50.585524 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:50.585495 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:03:50.586122 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:50.585623 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbrk8" podUID="9fa6e244-be92-4245-a54e-7aa303802f4b" Jul 29 04:03:51.585940 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:51.585542 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:03:51.585940 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:51.585682 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6vhn" podUID="350381e6-34a2-4f35-90d4-460ed24c7096" Jul 29 04:03:51.586480 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:51.586110 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:03:51.586480 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:51.586229 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9zpd" podUID="3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b" Jul 29 04:03:52.585412 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:52.585371 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:03:52.585585 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:52.585508 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbrk8" podUID="9fa6e244-be92-4245-a54e-7aa303802f4b" Jul 29 04:03:53.585806 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:53.585727 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:03:53.585806 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:53.585780 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:03:53.586310 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:53.585856 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9zpd" podUID="3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b" Jul 29 04:03:53.586310 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:53.585999 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6vhn" podUID="350381e6-34a2-4f35-90d4-460ed24c7096" Jul 29 04:03:54.585627 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:54.585596 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:03:54.585824 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:54.585713 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbrk8" podUID="9fa6e244-be92-4245-a54e-7aa303802f4b" Jul 29 04:03:55.585753 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:55.585719 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:03:55.585933 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:55.585856 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9zpd" podUID="3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b" Jul 29 04:03:55.586308 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:55.585932 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:03:55.586308 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:55.586076 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6vhn" podUID="350381e6-34a2-4f35-90d4-460ed24c7096" Jul 29 04:03:56.586291 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:56.586254 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:03:56.586731 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:56.586379 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbrk8" podUID="9fa6e244-be92-4245-a54e-7aa303802f4b" Jul 29 04:03:57.197549 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:57.197510 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fa6e244-be92-4245-a54e-7aa303802f4b-metrics-certs\") pod \"network-metrics-daemon-zbrk8\" (UID: \"9fa6e244-be92-4245-a54e-7aa303802f4b\") " pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:03:57.197830 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:57.197680 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 04:03:57.197830 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:57.197758 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fa6e244-be92-4245-a54e-7aa303802f4b-metrics-certs podName:9fa6e244-be92-4245-a54e-7aa303802f4b nodeName:}" failed. No retries permitted until 2026-07-29 04:04:13.197735756 +0000 UTC m=+34.057110481 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9fa6e244-be92-4245-a54e-7aa303802f4b-metrics-certs") pod "network-metrics-daemon-zbrk8" (UID: "9fa6e244-be92-4245-a54e-7aa303802f4b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 04:03:57.298227 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:57.298123 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b-original-pull-secret\") pod \"global-pull-secret-syncer-b9zpd\" (UID: \"3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b\") " pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:03:57.298391 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:57.298292 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 04:03:57.298391 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:57.298371 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b-original-pull-secret podName:3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b nodeName:}" failed. No retries permitted until 2026-07-29 04:04:13.298350798 +0000 UTC m=+34.157725544 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b-original-pull-secret") pod "global-pull-secret-syncer-b9zpd" (UID: "3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b") : object "kube-system"/"original-pull-secret" not registered Jul 29 04:03:57.398822 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:57.398779 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lqmq\" (UniqueName: \"kubernetes.io/projected/350381e6-34a2-4f35-90d4-460ed24c7096-kube-api-access-7lqmq\") pod \"network-check-target-x6vhn\" (UID: \"350381e6-34a2-4f35-90d4-460ed24c7096\") " pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:03:57.399014 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:57.398968 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 04:03:57.399014 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:57.398997 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 04:03:57.399014 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:57.399008 2579 projected.go:196] Error preparing data for projected volume kube-api-access-7lqmq for pod openshift-network-diagnostics/network-check-target-x6vhn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 04:03:57.399183 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:57.399077 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/350381e6-34a2-4f35-90d4-460ed24c7096-kube-api-access-7lqmq podName:350381e6-34a2-4f35-90d4-460ed24c7096 nodeName:}" failed. No retries permitted until 2026-07-29 04:04:13.399057535 +0000 UTC m=+34.258432258 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lqmq" (UniqueName: "kubernetes.io/projected/350381e6-34a2-4f35-90d4-460ed24c7096-kube-api-access-7lqmq") pod "network-check-target-x6vhn" (UID: "350381e6-34a2-4f35-90d4-460ed24c7096") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 04:03:57.586144 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:57.586055 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:03:57.586314 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:57.586059 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:03:57.586314 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:57.586199 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9zpd" podUID="3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b" Jul 29 04:03:57.586314 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:57.586284 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6vhn" podUID="350381e6-34a2-4f35-90d4-460ed24c7096" Jul 29 04:03:58.586091 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:58.586046 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:03:58.586303 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:58.586190 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbrk8" podUID="9fa6e244-be92-4245-a54e-7aa303802f4b" Jul 29 04:03:59.586472 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:59.586437 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:03:59.586933 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:59.586565 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9zpd" podUID="3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b" Jul 29 04:03:59.586933 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:03:59.586610 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:03:59.586933 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:03:59.586723 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6vhn" podUID="350381e6-34a2-4f35-90d4-460ed24c7096" Jul 29 04:04:00.586139 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:00.586100 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:04:00.586326 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:00.586243 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbrk8" podUID="9fa6e244-be92-4245-a54e-7aa303802f4b" Jul 29 04:04:01.586111 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:01.585933 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:04:01.586649 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:01.585933 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:04:01.586649 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:01.586222 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6vhn" podUID="350381e6-34a2-4f35-90d4-460ed24c7096" Jul 29 04:04:01.586649 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:01.586322 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9zpd" podUID="3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b" Jul 29 04:04:01.756283 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:01.756249 2579 generic.go:356] "Generic (PLEG): container finished" podID="a4d18741-fe36-4f1a-a290-795bfc955fb0" containerID="2e4cd70c9f09c11ee0ddd994456f5b051c0dc7dfacf7899a8ba3002af9ad293f" exitCode=0 Jul 29 04:04:01.756418 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:01.756335 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjpnt" event={"ID":"a4d18741-fe36-4f1a-a290-795bfc955fb0","Type":"ContainerDied","Data":"2e4cd70c9f09c11ee0ddd994456f5b051c0dc7dfacf7899a8ba3002af9ad293f"} Jul 29 04:04:01.757952 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:01.757927 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-492cr" event={"ID":"9384d535-332c-4320-a56e-a8cccbbbc5e8","Type":"ContainerStarted","Data":"2981c10085cb9dab3796faeae7bf00cfb97a180ef3992f75cb3f97d5f69af9a6"} Jul 29 04:04:01.759436 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:01.759412 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxzz5" event={"ID":"4f863b64-3884-435e-b469-06b94f479e3b","Type":"ContainerStarted","Data":"853de711311dcab96a8c39a80a3077a3297dfeae3a3dfa7e1b47baeaaeec5208"} Jul 29 04:04:01.760925 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:01.760899 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" event={"ID":"e5d96939-0ee9-454b-a70b-1031f1c38623","Type":"ContainerStarted","Data":"672297d0e2592c62f023be09b85af24cc335cba14da5e082cf14556444a08d71"} Jul 29 04:04:01.762217 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:01.762186 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v8d4n" event={"ID":"dc47e5bf-dc39-4b81-ba67-5c00203489e0","Type":"ContainerStarted","Data":"1137629baf7708412dadf1f9017b4b2612943c90e8038d57a7ae8935cab39e9c"} Jul 29 04:04:01.763545 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:01.763524 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wwg6z" event={"ID":"0055fae1-bd7e-4e87-834c-3a317313ae0c","Type":"ContainerStarted","Data":"4d6e180e88228d30c4a5f82191deb368feb176dc1921e70df0f36076a282be63"} Jul 29 04:04:01.766095 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:01.766064 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7792" event={"ID":"7fab7167-d61f-486e-85d2-58004704568e","Type":"ContainerStarted","Data":"1fb467e8a2b9e06df47c5acb7a935876431799c6232b0f3f8b2c4afde1bbf1aa"} Jul 29 04:04:01.766190 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:01.766122 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7792" event={"ID":"7fab7167-d61f-486e-85d2-58004704568e","Type":"ContainerStarted","Data":"04692482eba57143da0afad03a564ed7946705e6370cce9560e41707b79b1800"} Jul 29 04:04:01.766190 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:01.766157 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7792" event={"ID":"7fab7167-d61f-486e-85d2-58004704568e","Type":"ContainerStarted","Data":"575acc875468d965363bb05cb4f2635d5e3ff818f1f3d35acbb1f454dc547723"} Jul 29 04:04:01.766317 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:01.766200 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7792" event={"ID":"7fab7167-d61f-486e-85d2-58004704568e","Type":"ContainerStarted","Data":"4891e4483e0c18bddd0f924e20a9bf9d57d98bdf58e986ce690c32153592f77f"} Jul 29 04:04:01.766317 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:01.766245 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7792" event={"ID":"7fab7167-d61f-486e-85d2-58004704568e","Type":"ContainerStarted","Data":"e62f89a057ab0739a2f65e379cb41e38056e2019cf75a587fc1b9ab6e29cfc33"} Jul 29 04:04:01.767412 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:01.767395 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-snqz4" event={"ID":"ed77db79-64ab-4dda-8b0c-3ab2ec321512","Type":"ContainerStarted","Data":"aff2e45c0788c3960d66cec665092346ec899341c3468dd79144c2e24bfbb971"} Jul 29 04:04:01.799947 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:01.799893 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wwg6z" podStartSLOduration=4.358544659 podStartE2EDuration="22.799871708s" totalImagesPullingTime="18.441327049s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 04:03:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:04:01.799483863 +0000 UTC m=+22.658858613" watchObservedRunningTime="2026-07-29 04:04:01.799871708 +0000 UTC m=+22.659246433" Jul 29 04:04:01.825967 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:01.825917 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tf8wh" podStartSLOduration=4.38269112 podStartE2EDuration="22.825902137s" totalImagesPullingTime="18.443211017s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 04:03:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:04:01.825686293 +0000 UTC m=+22.685061017" watchObservedRunningTime="2026-07-29 04:04:01.825902137 +0000 UTC m=+22.685276862" Jul 29 04:04:01.841736 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:01.841637 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-snqz4" podStartSLOduration=4.302906551 podStartE2EDuration="22.841615378s" totalImagesPullingTime="18.538708827s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 04:03:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:04:01.840955547 +0000 UTC m=+22.700330284" watchObservedRunningTime="2026-07-29 04:04:01.841615378 +0000 UTC m=+22.700990103" Jul 29 04:04:01.854529 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:01.854467 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-v8d4n" podStartSLOduration=4.417077842 podStartE2EDuration="22.854447071s" totalImagesPullingTime="18.437369229s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 04:03:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:04:01.854298803 +0000 UTC m=+22.713673528" watchObservedRunningTime="2026-07-29 04:04:01.854447071 +0000 UTC m=+22.713821796" Jul 29 04:04:01.867742 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:01.867691 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-492cr" podStartSLOduration=8.540884928 podStartE2EDuration="22.867671725s" totalImagesPullingTime="14.326786797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:03:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:04:01.867433979 +0000 UTC m=+22.726808704" watchObservedRunningTime="2026-07-29 04:04:01.867671725 +0000 UTC m=+22.727046461" Jul 29 04:04:02.585642 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:02.585619 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:04:02.585785 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:02.585740 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbrk8" podUID="9fa6e244-be92-4245-a54e-7aa303802f4b" Jul 29 04:04:02.628805 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:02.628772 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 29 04:04:02.773057 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:02.772977 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7792" event={"ID":"7fab7167-d61f-486e-85d2-58004704568e","Type":"ContainerStarted","Data":"f31107c31ddab35f2369a335b3a902e3b424294c9136968f4528c7f51ba4f45a"} Jul 29 04:04:02.774596 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:02.774567 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2qc5b" event={"ID":"15922634-69e9-4620-ab51-f72a18853896","Type":"ContainerStarted","Data":"fb660731e817ecd99f8200f77a7b6d271ca68d33f5374a7f95d0139727945a59"} Jul 29 04:04:02.776563 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:02.776525 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxzz5" event={"ID":"4f863b64-3884-435e-b469-06b94f479e3b","Type":"ContainerStarted","Data":"2c8a43372d484d46079debc68c6e27e8a6a93fad2a5f8eaec7d74bebf8814bc9"} Jul 29 04:04:02.789650 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:02.789600 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2qc5b" podStartSLOduration=5.324293437 podStartE2EDuration="23.789582304s" totalImagesPullingTime="18.465288867s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 04:03:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:04:02.789193099 +0000 UTC m=+23.648567822" watchObservedRunningTime="2026-07-29 04:04:02.789582304 +0000 UTC m=+23.648957029" Jul 29 04:04:03.534237 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:03.534120 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T04:04:02.628796612Z","UUID":"3eb78fdb-5df6-4670-b0ec-1479ce0225e1","Handler":null,"Name":"","Endpoint":""} Jul 29 04:04:03.535959 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:03.535934 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 29 04:04:03.536101 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:03.535980 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 29 04:04:03.585901 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:03.585856 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:04:03.586060 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:03.585866 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:04:03.586060 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:03.585997 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6vhn" podUID="350381e6-34a2-4f35-90d4-460ed24c7096" Jul 29 04:04:03.586175 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:03.586061 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9zpd" podUID="3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b" Jul 29 04:04:03.780663 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:03.780626 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxzz5" event={"ID":"4f863b64-3884-435e-b469-06b94f479e3b","Type":"ContainerStarted","Data":"7209b02b5e81225d8eaf7c1fe7f4fbaa78ddf286d039080df057cef2ddee77ef"} Jul 29 04:04:03.800891 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:03.800793 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxzz5" podStartSLOduration=4.12957589 podStartE2EDuration="24.800778311s" totalImagesPullingTime="20.671202421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 04:03:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:04:03.800369674 +0000 UTC m=+24.659744411" watchObservedRunningTime="2026-07-29 04:04:03.800778311 +0000 UTC m=+24.660153035" Jul 29 04:04:04.182813 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:04.182621 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-492cr" Jul 29 04:04:04.585425 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:04.585353 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:04:04.585583 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:04.585483 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbrk8" podUID="9fa6e244-be92-4245-a54e-7aa303802f4b" Jul 29 04:04:04.786379 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:04.786345 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7792" event={"ID":"7fab7167-d61f-486e-85d2-58004704568e","Type":"ContainerStarted","Data":"6ea4b520beb6229dabcd31de4e5f4049cb5248a1af4fb5867aff1c1cdd0c5041"} Jul 29 04:04:05.585777 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:05.585739 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:04:05.585978 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:05.585789 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:04:05.585978 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:05.585876 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9zpd" podUID="3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b" Jul 29 04:04:05.586109 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:05.586006 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6vhn" podUID="350381e6-34a2-4f35-90d4-460ed24c7096" Jul 29 04:04:06.585790 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:06.585529 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:04:06.586456 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:06.585833 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbrk8" podUID="9fa6e244-be92-4245-a54e-7aa303802f4b" Jul 29 04:04:06.676819 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:06.676780 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-492cr" Jul 29 04:04:06.677453 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:06.677429 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-492cr" Jul 29 04:04:06.792808 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:06.792771 2579 generic.go:356] "Generic (PLEG): container finished" podID="a4d18741-fe36-4f1a-a290-795bfc955fb0" containerID="945acc1c039b9fa292fb2db6dac80ebc4e499205dbc25b747c45a0dfb052e602" exitCode=0 Jul 29 04:04:06.792965 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:06.792856 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjpnt" event={"ID":"a4d18741-fe36-4f1a-a290-795bfc955fb0","Type":"ContainerDied","Data":"945acc1c039b9fa292fb2db6dac80ebc4e499205dbc25b747c45a0dfb052e602"} Jul 29 04:04:06.796849 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:06.796821 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7792" event={"ID":"7fab7167-d61f-486e-85d2-58004704568e","Type":"ContainerStarted","Data":"fc397094f4a0b5215fdd45022da666c5ab5c35c5005744e315e52988f50707e3"} Jul 29 04:04:06.797494 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:06.797480 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-492cr" Jul 29 04:04:06.841357 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:06.841244 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-p7792" podStartSLOduration=9.316502081 podStartE2EDuration="27.841202023s" totalImagesPullingTime="18.524699942s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 04:03:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:04:06.840631216 +0000 UTC m=+27.700005932" watchObservedRunningTime="2026-07-29 04:04:06.841202023 +0000 UTC m=+27.700576747" Jul 29 04:04:07.585707 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:07.585666 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:04:07.585882 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:07.585685 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:04:07.585882 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:07.585824 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9zpd" podUID="3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b" Jul 29 04:04:07.585882 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:07.585845 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6vhn" podUID="350381e6-34a2-4f35-90d4-460ed24c7096" Jul 29 04:04:07.799544 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:07.799503 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:04:07.799727 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:07.799579 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:04:07.799911 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:07.799896 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:04:07.815649 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:07.815624 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:04:07.819711 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:07.819689 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:04:08.115428 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:08.115166 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x6vhn"] Jul 29 04:04:08.115621 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:08.115521 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:04:08.115687 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:08.115632 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6vhn" podUID="350381e6-34a2-4f35-90d4-460ed24c7096" Jul 29 04:04:08.118425 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:08.118397 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b9zpd"] Jul 29 04:04:08.118563 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:08.118544 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:04:08.118662 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:08.118643 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9zpd" podUID="3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b" Jul 29 04:04:08.119083 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:08.119059 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zbrk8"] Jul 29 04:04:08.119218 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:08.119169 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:04:08.119311 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:08.119286 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbrk8" podUID="9fa6e244-be92-4245-a54e-7aa303802f4b" Jul 29 04:04:08.803142 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:08.803111 2579 generic.go:356] "Generic (PLEG): container finished" podID="a4d18741-fe36-4f1a-a290-795bfc955fb0" containerID="6f3580e849c195e6b11e713254047df59a052f9802e18c9919b91792090c0ebc" exitCode=0 Jul 29 04:04:08.803558 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:08.803190 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjpnt" event={"ID":"a4d18741-fe36-4f1a-a290-795bfc955fb0","Type":"ContainerDied","Data":"6f3580e849c195e6b11e713254047df59a052f9802e18c9919b91792090c0ebc"} Jul 29 04:04:09.585918 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:09.585884 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:04:09.586060 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:09.585972 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9zpd" podUID="3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b" Jul 29 04:04:09.586060 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:09.586049 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:04:09.586165 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:09.586148 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6vhn" podUID="350381e6-34a2-4f35-90d4-460ed24c7096" Jul 29 04:04:09.586239 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:09.586228 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:04:09.586304 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:09.586291 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbrk8" podUID="9fa6e244-be92-4245-a54e-7aa303802f4b" Jul 29 04:04:09.807484 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:09.807452 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjpnt" event={"ID":"a4d18741-fe36-4f1a-a290-795bfc955fb0","Type":"ContainerStarted","Data":"1fe8dfd49f41f0220cc776f48a170dc00a4a15628e8bd6129ea28eab36e20144"} Jul 29 04:04:10.811726 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:10.811692 2579 generic.go:356] "Generic (PLEG): container finished" podID="a4d18741-fe36-4f1a-a290-795bfc955fb0" containerID="1fe8dfd49f41f0220cc776f48a170dc00a4a15628e8bd6129ea28eab36e20144" exitCode=0 Jul 29 04:04:10.812286 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:10.811751 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjpnt" event={"ID":"a4d18741-fe36-4f1a-a290-795bfc955fb0","Type":"ContainerDied","Data":"1fe8dfd49f41f0220cc776f48a170dc00a4a15628e8bd6129ea28eab36e20144"} Jul 29 04:04:11.585802 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:11.585768 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:04:11.585977 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:11.585774 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:04:11.585977 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:11.585910 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbrk8" podUID="9fa6e244-be92-4245-a54e-7aa303802f4b" Jul 29 04:04:11.585977 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:11.585966 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9zpd" podUID="3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b" Jul 29 04:04:11.586100 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:11.585775 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:04:11.586100 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:11.586038 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6vhn" podUID="350381e6-34a2-4f35-90d4-460ed24c7096" Jul 29 04:04:13.222311 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:13.222269 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fa6e244-be92-4245-a54e-7aa303802f4b-metrics-certs\") pod \"network-metrics-daemon-zbrk8\" (UID: \"9fa6e244-be92-4245-a54e-7aa303802f4b\") " pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:04:13.222760 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:13.222428 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 04:04:13.222760 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:13.222492 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fa6e244-be92-4245-a54e-7aa303802f4b-metrics-certs podName:9fa6e244-be92-4245-a54e-7aa303802f4b nodeName:}" failed. No retries permitted until 2026-07-29 04:04:45.222473049 +0000 UTC m=+66.081847773 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9fa6e244-be92-4245-a54e-7aa303802f4b-metrics-certs") pod "network-metrics-daemon-zbrk8" (UID: "9fa6e244-be92-4245-a54e-7aa303802f4b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 04:04:13.326304 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:13.324408 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b-original-pull-secret\") pod \"global-pull-secret-syncer-b9zpd\" (UID: \"3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b\") " pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:04:13.326304 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:13.324581 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 04:04:13.326304 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:13.324655 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b-original-pull-secret podName:3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b nodeName:}" failed. No retries permitted until 2026-07-29 04:04:45.324634533 +0000 UTC m=+66.184009248 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b-original-pull-secret") pod "global-pull-secret-syncer-b9zpd" (UID: "3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b") : object "kube-system"/"original-pull-secret" not registered Jul 29 04:04:13.425638 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:13.425405 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lqmq\" (UniqueName: \"kubernetes.io/projected/350381e6-34a2-4f35-90d4-460ed24c7096-kube-api-access-7lqmq\") pod \"network-check-target-x6vhn\" (UID: \"350381e6-34a2-4f35-90d4-460ed24c7096\") " pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:04:13.425817 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:13.425588 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 04:04:13.425817 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:13.425697 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 04:04:13.425817 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:13.425712 2579 projected.go:196] Error preparing data for projected volume kube-api-access-7lqmq for pod openshift-network-diagnostics/network-check-target-x6vhn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 04:04:13.425817 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:13.425782 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/350381e6-34a2-4f35-90d4-460ed24c7096-kube-api-access-7lqmq podName:350381e6-34a2-4f35-90d4-460ed24c7096 nodeName:}" failed. No retries permitted until 2026-07-29 04:04:45.425763435 +0000 UTC m=+66.285138157 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lqmq" (UniqueName: "kubernetes.io/projected/350381e6-34a2-4f35-90d4-460ed24c7096-kube-api-access-7lqmq") pod "network-check-target-x6vhn" (UID: "350381e6-34a2-4f35-90d4-460ed24c7096") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 04:04:13.590084 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:13.590054 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:04:13.590259 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:13.590169 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:04:13.590259 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:13.590183 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6vhn" podUID="350381e6-34a2-4f35-90d4-460ed24c7096" Jul 29 04:04:13.590259 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:13.590219 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:04:13.590431 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:13.590311 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbrk8" podUID="9fa6e244-be92-4245-a54e-7aa303802f4b" Jul 29 04:04:13.590431 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:13.590391 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9zpd" podUID="3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b" Jul 29 04:04:14.022161 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.022117 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-111.ec2.internal" event="NodeReady" Jul 29 04:04:14.022332 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.022298 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 29 04:04:14.064893 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.064847 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lfjbp"] Jul 29 04:04:14.070174 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.070146 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bprlw"] Jul 29 04:04:14.070301 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.070280 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lfjbp" Jul 29 04:04:14.072461 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.072436 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7m6rk\"" Jul 29 04:04:14.072571 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.072477 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 29 04:04:14.072571 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.072498 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 29 04:04:14.073812 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.073635 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-55d9f6d798-l5rf7"] Jul 29 04:04:14.073812 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.073771 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bprlw" Jul 29 04:04:14.075952 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.075932 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 29 04:04:14.076235 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.076198 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 29 04:04:14.076343 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.076280 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-p8dgx\"" Jul 29 04:04:14.079711 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.077179 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 29 04:04:14.079711 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.077715 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bprlw"] Jul 29 04:04:14.079711 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.077806 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:14.079711 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.078359 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lfjbp"] Jul 29 04:04:14.079711 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.078912 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55d9f6d798-l5rf7"] Jul 29 04:04:14.080372 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.080127 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 29 04:04:14.080372 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.080161 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 29 04:04:14.080372 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.080173 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 29 04:04:14.080372 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.080328 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qmdc2\"" Jul 29 04:04:14.085996 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.085974 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 29 04:04:14.232539 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.232505 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcdad901-fa84-42eb-8e6d-70c86d2639cb-cert\") pod \"ingress-canary-bprlw\" (UID: \"dcdad901-fa84-42eb-8e6d-70c86d2639cb\") " pod="openshift-ingress-canary/ingress-canary-bprlw" Jul 29 04:04:14.233005 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.232566 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-registry-tls\") pod \"image-registry-55d9f6d798-l5rf7\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:14.233005 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.232588 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f194f797-9d02-465f-9293-e5e6d4226a21-registry-certificates\") pod \"image-registry-55d9f6d798-l5rf7\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:14.233005 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.232645 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnsfd\" (UniqueName: \"kubernetes.io/projected/ea98d97f-f580-434b-8d20-b687c0d951fd-kube-api-access-pnsfd\") pod \"dns-default-lfjbp\" (UID: \"ea98d97f-f580-434b-8d20-b687c0d951fd\") " pod="openshift-dns/dns-default-lfjbp" Jul 29 04:04:14.233005 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.232669 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvddl\" (UniqueName: \"kubernetes.io/projected/dcdad901-fa84-42eb-8e6d-70c86d2639cb-kube-api-access-gvddl\") pod \"ingress-canary-bprlw\" (UID: \"dcdad901-fa84-42eb-8e6d-70c86d2639cb\") " pod="openshift-ingress-canary/ingress-canary-bprlw" Jul 29 04:04:14.233005 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.232715 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f194f797-9d02-465f-9293-e5e6d4226a21-trusted-ca\") pod \"image-registry-55d9f6d798-l5rf7\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:14.233005 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.232732 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f194f797-9d02-465f-9293-e5e6d4226a21-installation-pull-secrets\") pod \"image-registry-55d9f6d798-l5rf7\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:14.233005 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.232755 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-bound-sa-token\") pod \"image-registry-55d9f6d798-l5rf7\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:14.233005 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.232808 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ea98d97f-f580-434b-8d20-b687c0d951fd-config-volume\") pod \"dns-default-lfjbp\" (UID: \"ea98d97f-f580-434b-8d20-b687c0d951fd\") " pod="openshift-dns/dns-default-lfjbp" Jul 29 04:04:14.233005 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.232848 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f194f797-9d02-465f-9293-e5e6d4226a21-ca-trust-extracted\") pod \"image-registry-55d9f6d798-l5rf7\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:14.233005 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.232872 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4pbf\" (UniqueName: \"kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-kube-api-access-b4pbf\") pod \"image-registry-55d9f6d798-l5rf7\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:14.233005 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.232891 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea98d97f-f580-434b-8d20-b687c0d951fd-metrics-tls\") pod \"dns-default-lfjbp\" (UID: \"ea98d97f-f580-434b-8d20-b687c0d951fd\") " pod="openshift-dns/dns-default-lfjbp" Jul 29 04:04:14.233005 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.232919 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea98d97f-f580-434b-8d20-b687c0d951fd-tmp-dir\") pod \"dns-default-lfjbp\" (UID: \"ea98d97f-f580-434b-8d20-b687c0d951fd\") " pod="openshift-dns/dns-default-lfjbp" Jul 29 04:04:14.233005 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.232966 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f194f797-9d02-465f-9293-e5e6d4226a21-image-registry-private-configuration\") pod \"image-registry-55d9f6d798-l5rf7\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:14.333962 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.333871 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f194f797-9d02-465f-9293-e5e6d4226a21-image-registry-private-configuration\") pod \"image-registry-55d9f6d798-l5rf7\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:14.333962 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.333924 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcdad901-fa84-42eb-8e6d-70c86d2639cb-cert\") pod \"ingress-canary-bprlw\" (UID: \"dcdad901-fa84-42eb-8e6d-70c86d2639cb\") " pod="openshift-ingress-canary/ingress-canary-bprlw" Jul 29 04:04:14.333962 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.333965 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-registry-tls\") pod \"image-registry-55d9f6d798-l5rf7\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:14.334286 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.333999 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f194f797-9d02-465f-9293-e5e6d4226a21-registry-certificates\") pod \"image-registry-55d9f6d798-l5rf7\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:14.334286 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.334040 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnsfd\" (UniqueName: \"kubernetes.io/projected/ea98d97f-f580-434b-8d20-b687c0d951fd-kube-api-access-pnsfd\") pod \"dns-default-lfjbp\" (UID: \"ea98d97f-f580-434b-8d20-b687c0d951fd\") " pod="openshift-dns/dns-default-lfjbp" Jul 29 04:04:14.334286 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.334061 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvddl\" (UniqueName: \"kubernetes.io/projected/dcdad901-fa84-42eb-8e6d-70c86d2639cb-kube-api-access-gvddl\") pod \"ingress-canary-bprlw\" (UID: \"dcdad901-fa84-42eb-8e6d-70c86d2639cb\") " pod="openshift-ingress-canary/ingress-canary-bprlw" Jul 29 04:04:14.334286 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.334087 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f194f797-9d02-465f-9293-e5e6d4226a21-trusted-ca\") pod \"image-registry-55d9f6d798-l5rf7\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:14.334286 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.334112 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f194f797-9d02-465f-9293-e5e6d4226a21-installation-pull-secrets\") pod \"image-registry-55d9f6d798-l5rf7\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:14.334286 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.334159 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-bound-sa-token\") pod \"image-registry-55d9f6d798-l5rf7\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:14.334286 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.334176 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ea98d97f-f580-434b-8d20-b687c0d951fd-config-volume\") pod \"dns-default-lfjbp\" (UID: \"ea98d97f-f580-434b-8d20-b687c0d951fd\") " pod="openshift-dns/dns-default-lfjbp" Jul 29 04:04:14.334286 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.334191 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f194f797-9d02-465f-9293-e5e6d4226a21-ca-trust-extracted\") pod \"image-registry-55d9f6d798-l5rf7\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:14.334286 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.334229 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4pbf\" (UniqueName: \"kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-kube-api-access-b4pbf\") pod \"image-registry-55d9f6d798-l5rf7\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:14.334286 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.334252 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea98d97f-f580-434b-8d20-b687c0d951fd-metrics-tls\") pod \"dns-default-lfjbp\" (UID: \"ea98d97f-f580-434b-8d20-b687c0d951fd\") " pod="openshift-dns/dns-default-lfjbp" Jul 29 04:04:14.334286 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.334271 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea98d97f-f580-434b-8d20-b687c0d951fd-tmp-dir\") pod \"dns-default-lfjbp\" (UID: \"ea98d97f-f580-434b-8d20-b687c0d951fd\") " pod="openshift-dns/dns-default-lfjbp" Jul 29 04:04:14.334743 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.334580 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea98d97f-f580-434b-8d20-b687c0d951fd-tmp-dir\") pod \"dns-default-lfjbp\" (UID: \"ea98d97f-f580-434b-8d20-b687c0d951fd\") " pod="openshift-dns/dns-default-lfjbp" Jul 29 04:04:14.334896 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.334874 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f194f797-9d02-465f-9293-e5e6d4226a21-registry-certificates\") pod \"image-registry-55d9f6d798-l5rf7\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:14.334988 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:14.334089 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 04:04:14.334988 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:14.334916 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55d9f6d798-l5rf7: secret "image-registry-tls" not found Jul 29 04:04:14.334988 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:14.334955 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-registry-tls podName:f194f797-9d02-465f-9293-e5e6d4226a21 nodeName:}" failed. No retries permitted until 2026-07-29 04:04:14.834941009 +0000 UTC m=+35.694315725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-registry-tls") pod "image-registry-55d9f6d798-l5rf7" (UID: "f194f797-9d02-465f-9293-e5e6d4226a21") : secret "image-registry-tls" not found Jul 29 04:04:14.335673 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.335348 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ea98d97f-f580-434b-8d20-b687c0d951fd-config-volume\") pod \"dns-default-lfjbp\" (UID: \"ea98d97f-f580-434b-8d20-b687c0d951fd\") " pod="openshift-dns/dns-default-lfjbp" Jul 29 04:04:14.335673 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:14.335454 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 04:04:14.335673 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:14.335500 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea98d97f-f580-434b-8d20-b687c0d951fd-metrics-tls podName:ea98d97f-f580-434b-8d20-b687c0d951fd nodeName:}" failed. No retries permitted until 2026-07-29 04:04:14.835485812 +0000 UTC m=+35.694860536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea98d97f-f580-434b-8d20-b687c0d951fd-metrics-tls") pod "dns-default-lfjbp" (UID: "ea98d97f-f580-434b-8d20-b687c0d951fd") : secret "dns-default-metrics-tls" not found Jul 29 04:04:14.335673 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.335621 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f194f797-9d02-465f-9293-e5e6d4226a21-ca-trust-extracted\") pod \"image-registry-55d9f6d798-l5rf7\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:14.335673 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.335627 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f194f797-9d02-465f-9293-e5e6d4226a21-trusted-ca\") pod \"image-registry-55d9f6d798-l5rf7\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:14.338762 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.338738 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f194f797-9d02-465f-9293-e5e6d4226a21-installation-pull-secrets\") pod \"image-registry-55d9f6d798-l5rf7\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:14.338879 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.338757 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f194f797-9d02-465f-9293-e5e6d4226a21-image-registry-private-configuration\") pod \"image-registry-55d9f6d798-l5rf7\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:14.338879 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.338837 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcdad901-fa84-42eb-8e6d-70c86d2639cb-cert\") pod \"ingress-canary-bprlw\" (UID: \"dcdad901-fa84-42eb-8e6d-70c86d2639cb\") " pod="openshift-ingress-canary/ingress-canary-bprlw" Jul 29 04:04:14.342920 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.342897 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnsfd\" (UniqueName: \"kubernetes.io/projected/ea98d97f-f580-434b-8d20-b687c0d951fd-kube-api-access-pnsfd\") pod \"dns-default-lfjbp\" (UID: \"ea98d97f-f580-434b-8d20-b687c0d951fd\") " pod="openshift-dns/dns-default-lfjbp" Jul 29 04:04:14.343230 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.343189 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-bound-sa-token\") pod \"image-registry-55d9f6d798-l5rf7\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:14.343464 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.343418 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvddl\" (UniqueName: \"kubernetes.io/projected/dcdad901-fa84-42eb-8e6d-70c86d2639cb-kube-api-access-gvddl\") pod \"ingress-canary-bprlw\" (UID: \"dcdad901-fa84-42eb-8e6d-70c86d2639cb\") " pod="openshift-ingress-canary/ingress-canary-bprlw" Jul 29 04:04:14.344060 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.344042 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4pbf\" (UniqueName: \"kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-kube-api-access-b4pbf\") pod \"image-registry-55d9f6d798-l5rf7\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:14.394376 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.394341 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bprlw" Jul 29 04:04:14.838740 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.838701 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-registry-tls\") pod \"image-registry-55d9f6d798-l5rf7\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:14.838953 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:14.838800 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea98d97f-f580-434b-8d20-b687c0d951fd-metrics-tls\") pod \"dns-default-lfjbp\" (UID: \"ea98d97f-f580-434b-8d20-b687c0d951fd\") " pod="openshift-dns/dns-default-lfjbp" Jul 29 04:04:14.838953 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:14.838893 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 04:04:14.838953 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:14.838914 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55d9f6d798-l5rf7: secret "image-registry-tls" not found Jul 29 04:04:14.838953 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:14.838914 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 04:04:14.839121 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:14.838984 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-registry-tls podName:f194f797-9d02-465f-9293-e5e6d4226a21 nodeName:}" failed. No retries permitted until 2026-07-29 04:04:15.83896354 +0000 UTC m=+36.698338243 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-registry-tls") pod "image-registry-55d9f6d798-l5rf7" (UID: "f194f797-9d02-465f-9293-e5e6d4226a21") : secret "image-registry-tls" not found Jul 29 04:04:14.839121 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:14.839006 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea98d97f-f580-434b-8d20-b687c0d951fd-metrics-tls podName:ea98d97f-f580-434b-8d20-b687c0d951fd nodeName:}" failed. No retries permitted until 2026-07-29 04:04:15.838996174 +0000 UTC m=+36.698370908 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea98d97f-f580-434b-8d20-b687c0d951fd-metrics-tls") pod "dns-default-lfjbp" (UID: "ea98d97f-f580-434b-8d20-b687c0d951fd") : secret "dns-default-metrics-tls" not found Jul 29 04:04:15.008744 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:15.008718 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bprlw"] Jul 29 04:04:15.011724 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:04:15.011693 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcdad901_fa84_42eb_8e6d_70c86d2639cb.slice/crio-31542aea4557147f249a04415fc3e092633f724dedb37633050cbe30a9142a27 WatchSource:0}: Error finding container 31542aea4557147f249a04415fc3e092633f724dedb37633050cbe30a9142a27: Status 404 returned error can't find the container with id 31542aea4557147f249a04415fc3e092633f724dedb37633050cbe30a9142a27 Jul 29 04:04:15.585494 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:15.585307 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:04:15.585928 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:15.585307 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:04:15.585928 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:15.585307 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:04:15.588315 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:15.588292 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 04:04:15.588428 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:15.588295 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 04:04:15.589366 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:15.589345 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 04:04:15.589509 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:15.589451 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bzqwr\"" Jul 29 04:04:15.589509 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:15.589460 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mrbfg\"" Jul 29 04:04:15.589509 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:15.589467 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 04:04:15.827576 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:15.827544 2579 generic.go:356] "Generic (PLEG): container finished" podID="a4d18741-fe36-4f1a-a290-795bfc955fb0" containerID="46ec06dbb7a496f2d3d40e3d77039291e9db248731350483ae8f7b6f4dc43464" exitCode=0 Jul 29 04:04:15.827753 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:15.827627 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjpnt" event={"ID":"a4d18741-fe36-4f1a-a290-795bfc955fb0","Type":"ContainerDied","Data":"46ec06dbb7a496f2d3d40e3d77039291e9db248731350483ae8f7b6f4dc43464"} Jul 29 04:04:15.831040 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:15.831005 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bprlw" event={"ID":"dcdad901-fa84-42eb-8e6d-70c86d2639cb","Type":"ContainerStarted","Data":"31542aea4557147f249a04415fc3e092633f724dedb37633050cbe30a9142a27"} Jul 29 04:04:15.846621 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:15.846599 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea98d97f-f580-434b-8d20-b687c0d951fd-metrics-tls\") pod \"dns-default-lfjbp\" (UID: \"ea98d97f-f580-434b-8d20-b687c0d951fd\") " pod="openshift-dns/dns-default-lfjbp" Jul 29 04:04:15.846740 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:15.846687 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-registry-tls\") pod \"image-registry-55d9f6d798-l5rf7\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:15.846807 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:15.846754 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 04:04:15.846860 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:15.846809 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 04:04:15.846860 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:15.846821 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea98d97f-f580-434b-8d20-b687c0d951fd-metrics-tls podName:ea98d97f-f580-434b-8d20-b687c0d951fd nodeName:}" failed. No retries permitted until 2026-07-29 04:04:17.846802264 +0000 UTC m=+38.706176968 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea98d97f-f580-434b-8d20-b687c0d951fd-metrics-tls") pod "dns-default-lfjbp" (UID: "ea98d97f-f580-434b-8d20-b687c0d951fd") : secret "dns-default-metrics-tls" not found Jul 29 04:04:15.846860 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:15.846824 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55d9f6d798-l5rf7: secret "image-registry-tls" not found Jul 29 04:04:15.846998 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:15.846873 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-registry-tls podName:f194f797-9d02-465f-9293-e5e6d4226a21 nodeName:}" failed. No retries permitted until 2026-07-29 04:04:17.846857666 +0000 UTC m=+38.706232382 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-registry-tls") pod "image-registry-55d9f6d798-l5rf7" (UID: "f194f797-9d02-465f-9293-e5e6d4226a21") : secret "image-registry-tls" not found Jul 29 04:04:15.893585 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:15.893553 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-c8mlp"] Jul 29 04:04:15.898574 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:15.898553 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-c8mlp" Jul 29 04:04:15.900969 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:15.900942 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 29 04:04:15.900969 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:15.900952 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 29 04:04:15.901137 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:15.901032 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-qzk7b\"" Jul 29 04:04:15.909078 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:15.909057 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-c8mlp"] Jul 29 04:04:16.048890 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:16.048853 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppqbm\" (UniqueName: \"kubernetes.io/projected/2dafc4cf-b1a5-4907-8f24-a30ca4f08b30-kube-api-access-ppqbm\") pod \"migrator-6865c9967-c8mlp\" (UID: \"2dafc4cf-b1a5-4907-8f24-a30ca4f08b30\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-c8mlp" Jul 29 04:04:16.150317 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:16.150230 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppqbm\" (UniqueName: \"kubernetes.io/projected/2dafc4cf-b1a5-4907-8f24-a30ca4f08b30-kube-api-access-ppqbm\") pod \"migrator-6865c9967-c8mlp\" (UID: \"2dafc4cf-b1a5-4907-8f24-a30ca4f08b30\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-c8mlp" Jul 29 04:04:16.159924 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:16.159896 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppqbm\" (UniqueName: \"kubernetes.io/projected/2dafc4cf-b1a5-4907-8f24-a30ca4f08b30-kube-api-access-ppqbm\") pod \"migrator-6865c9967-c8mlp\" (UID: \"2dafc4cf-b1a5-4907-8f24-a30ca4f08b30\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-c8mlp" Jul 29 04:04:16.218998 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:16.218963 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-c8mlp" Jul 29 04:04:16.302958 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:16.300325 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-56kwm"] Jul 29 04:04:16.308006 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:16.306359 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-56kwm" Jul 29 04:04:16.309341 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:16.309311 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 29 04:04:16.309841 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:16.309757 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2xj8k\"" Jul 29 04:04:16.310033 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:16.309997 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 29 04:04:16.310541 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:16.310319 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 29 04:04:16.310541 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:16.310334 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 29 04:04:16.314022 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:16.313998 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-56kwm"] Jul 29 04:04:16.371289 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:16.371258 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-c8mlp"] Jul 29 04:04:16.375082 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:04:16.375056 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dafc4cf_b1a5_4907_8f24_a30ca4f08b30.slice/crio-2a345a57faedb13e11ffdd5c5a9cf5d0a26f2149278d3cdb7820b6624b4b2d2b WatchSource:0}: Error finding container 2a345a57faedb13e11ffdd5c5a9cf5d0a26f2149278d3cdb7820b6624b4b2d2b: Status 404 returned error can't find the container with id 2a345a57faedb13e11ffdd5c5a9cf5d0a26f2149278d3cdb7820b6624b4b2d2b Jul 29 04:04:16.453242 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:16.453187 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sxz7\" (UniqueName: \"kubernetes.io/projected/1deee91e-bb4a-4511-953d-9fd13697bdc1-kube-api-access-4sxz7\") pod \"insights-runtime-extractor-56kwm\" (UID: \"1deee91e-bb4a-4511-953d-9fd13697bdc1\") " pod="openshift-insights/insights-runtime-extractor-56kwm" Jul 29 04:04:16.453437 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:16.453305 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1deee91e-bb4a-4511-953d-9fd13697bdc1-data-volume\") pod \"insights-runtime-extractor-56kwm\" (UID: \"1deee91e-bb4a-4511-953d-9fd13697bdc1\") " pod="openshift-insights/insights-runtime-extractor-56kwm" Jul 29 04:04:16.453437 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:16.453424 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1deee91e-bb4a-4511-953d-9fd13697bdc1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-56kwm\" (UID: \"1deee91e-bb4a-4511-953d-9fd13697bdc1\") " pod="openshift-insights/insights-runtime-extractor-56kwm" Jul 29 04:04:16.453555 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:16.453481 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1deee91e-bb4a-4511-953d-9fd13697bdc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-56kwm\" (UID: \"1deee91e-bb4a-4511-953d-9fd13697bdc1\") " pod="openshift-insights/insights-runtime-extractor-56kwm" Jul 29 04:04:16.453555 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:16.453539 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1deee91e-bb4a-4511-953d-9fd13697bdc1-crio-socket\") pod \"insights-runtime-extractor-56kwm\" (UID: \"1deee91e-bb4a-4511-953d-9fd13697bdc1\") " pod="openshift-insights/insights-runtime-extractor-56kwm" Jul 29 04:04:16.554414 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:16.554380 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1deee91e-bb4a-4511-953d-9fd13697bdc1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-56kwm\" (UID: \"1deee91e-bb4a-4511-953d-9fd13697bdc1\") " pod="openshift-insights/insights-runtime-extractor-56kwm" Jul 29 04:04:16.557545 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:16.554812 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 04:04:16.557545 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:16.554908 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1deee91e-bb4a-4511-953d-9fd13697bdc1-kube-rbac-proxy-cm podName:1deee91e-bb4a-4511-953d-9fd13697bdc1 nodeName:}" failed. No retries permitted until 2026-07-29 04:04:17.054886559 +0000 UTC m=+37.914261284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1deee91e-bb4a-4511-953d-9fd13697bdc1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-56kwm" (UID: "1deee91e-bb4a-4511-953d-9fd13697bdc1") : configmap "kube-rbac-proxy" not found Jul 29 04:04:16.557545 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:16.554912 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 04:04:16.557545 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:16.554990 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1deee91e-bb4a-4511-953d-9fd13697bdc1-insights-runtime-extractor-tls podName:1deee91e-bb4a-4511-953d-9fd13697bdc1 nodeName:}" failed. No retries permitted until 2026-07-29 04:04:17.054971044 +0000 UTC m=+37.914345766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1deee91e-bb4a-4511-953d-9fd13697bdc1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-56kwm" (UID: "1deee91e-bb4a-4511-953d-9fd13697bdc1") : secret "insights-runtime-extractor-tls" not found Jul 29 04:04:16.557545 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:16.554724 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1deee91e-bb4a-4511-953d-9fd13697bdc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-56kwm\" (UID: \"1deee91e-bb4a-4511-953d-9fd13697bdc1\") " pod="openshift-insights/insights-runtime-extractor-56kwm" Jul 29 04:04:16.557545 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:16.555065 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1deee91e-bb4a-4511-953d-9fd13697bdc1-crio-socket\") pod \"insights-runtime-extractor-56kwm\" (UID: \"1deee91e-bb4a-4511-953d-9fd13697bdc1\") " pod="openshift-insights/insights-runtime-extractor-56kwm" Jul 29 04:04:16.557545 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:16.555123 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sxz7\" (UniqueName: \"kubernetes.io/projected/1deee91e-bb4a-4511-953d-9fd13697bdc1-kube-api-access-4sxz7\") pod \"insights-runtime-extractor-56kwm\" (UID: \"1deee91e-bb4a-4511-953d-9fd13697bdc1\") " pod="openshift-insights/insights-runtime-extractor-56kwm" Jul 29 04:04:16.557545 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:16.555162 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1deee91e-bb4a-4511-953d-9fd13697bdc1-data-volume\") pod \"insights-runtime-extractor-56kwm\" (UID: \"1deee91e-bb4a-4511-953d-9fd13697bdc1\") " pod="openshift-insights/insights-runtime-extractor-56kwm" Jul 29 04:04:16.557545 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:16.555337 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1deee91e-bb4a-4511-953d-9fd13697bdc1-crio-socket\") pod \"insights-runtime-extractor-56kwm\" (UID: \"1deee91e-bb4a-4511-953d-9fd13697bdc1\") " pod="openshift-insights/insights-runtime-extractor-56kwm" Jul 29 04:04:16.557545 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:16.555671 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1deee91e-bb4a-4511-953d-9fd13697bdc1-data-volume\") pod \"insights-runtime-extractor-56kwm\" (UID: \"1deee91e-bb4a-4511-953d-9fd13697bdc1\") " pod="openshift-insights/insights-runtime-extractor-56kwm" Jul 29 04:04:16.565630 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:16.565600 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sxz7\" (UniqueName: \"kubernetes.io/projected/1deee91e-bb4a-4511-953d-9fd13697bdc1-kube-api-access-4sxz7\") pod \"insights-runtime-extractor-56kwm\" (UID: \"1deee91e-bb4a-4511-953d-9fd13697bdc1\") " pod="openshift-insights/insights-runtime-extractor-56kwm" Jul 29 04:04:16.840021 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:16.838069 2579 generic.go:356] "Generic (PLEG): container finished" podID="a4d18741-fe36-4f1a-a290-795bfc955fb0" containerID="7e072c8febf0cd9c674c2350cf54d57e6f0143df8131d8a101b83be4798afe47" exitCode=0 Jul 29 04:04:16.840021 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:16.838189 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjpnt" event={"ID":"a4d18741-fe36-4f1a-a290-795bfc955fb0","Type":"ContainerDied","Data":"7e072c8febf0cd9c674c2350cf54d57e6f0143df8131d8a101b83be4798afe47"} Jul 29 04:04:16.840021 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:16.839877 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-c8mlp" event={"ID":"2dafc4cf-b1a5-4907-8f24-a30ca4f08b30","Type":"ContainerStarted","Data":"2a345a57faedb13e11ffdd5c5a9cf5d0a26f2149278d3cdb7820b6624b4b2d2b"} Jul 29 04:04:17.059221 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:17.059175 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1deee91e-bb4a-4511-953d-9fd13697bdc1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-56kwm\" (UID: \"1deee91e-bb4a-4511-953d-9fd13697bdc1\") " pod="openshift-insights/insights-runtime-extractor-56kwm" Jul 29 04:04:17.059367 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:17.059309 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 04:04:17.059418 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:17.059376 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1deee91e-bb4a-4511-953d-9fd13697bdc1-kube-rbac-proxy-cm podName:1deee91e-bb4a-4511-953d-9fd13697bdc1 nodeName:}" failed. No retries permitted until 2026-07-29 04:04:18.059359704 +0000 UTC m=+38.918734434 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1deee91e-bb4a-4511-953d-9fd13697bdc1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-56kwm" (UID: "1deee91e-bb4a-4511-953d-9fd13697bdc1") : configmap "kube-rbac-proxy" not found Jul 29 04:04:17.059557 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:17.059532 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1deee91e-bb4a-4511-953d-9fd13697bdc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-56kwm\" (UID: \"1deee91e-bb4a-4511-953d-9fd13697bdc1\") " pod="openshift-insights/insights-runtime-extractor-56kwm" Jul 29 04:04:17.059707 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:17.059688 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 04:04:17.059768 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:17.059757 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1deee91e-bb4a-4511-953d-9fd13697bdc1-insights-runtime-extractor-tls podName:1deee91e-bb4a-4511-953d-9fd13697bdc1 nodeName:}" failed. No retries permitted until 2026-07-29 04:04:18.059738919 +0000 UTC m=+38.919113629 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1deee91e-bb4a-4511-953d-9fd13697bdc1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-56kwm" (UID: "1deee91e-bb4a-4511-953d-9fd13697bdc1") : secret "insights-runtime-extractor-tls" not found Jul 29 04:04:17.845877 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:17.845799 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjpnt" event={"ID":"a4d18741-fe36-4f1a-a290-795bfc955fb0","Type":"ContainerStarted","Data":"33048f301c443768dec79ccabcc440f54ca584e1a65e059fb655aa8bba577c43"} Jul 29 04:04:17.867387 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:17.867352 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea98d97f-f580-434b-8d20-b687c0d951fd-metrics-tls\") pod \"dns-default-lfjbp\" (UID: \"ea98d97f-f580-434b-8d20-b687c0d951fd\") " pod="openshift-dns/dns-default-lfjbp" Jul 29 04:04:17.867534 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:17.867429 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-registry-tls\") pod \"image-registry-55d9f6d798-l5rf7\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:17.867534 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:17.867511 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 04:04:17.867605 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:17.867543 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 04:04:17.867605 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:17.867558 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55d9f6d798-l5rf7: secret "image-registry-tls" not found Jul 29 04:04:17.867605 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:17.867584 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea98d97f-f580-434b-8d20-b687c0d951fd-metrics-tls podName:ea98d97f-f580-434b-8d20-b687c0d951fd nodeName:}" failed. No retries permitted until 2026-07-29 04:04:21.867563557 +0000 UTC m=+42.726938282 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea98d97f-f580-434b-8d20-b687c0d951fd-metrics-tls") pod "dns-default-lfjbp" (UID: "ea98d97f-f580-434b-8d20-b687c0d951fd") : secret "dns-default-metrics-tls" not found Jul 29 04:04:17.867605 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:17.867600 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-registry-tls podName:f194f797-9d02-465f-9293-e5e6d4226a21 nodeName:}" failed. No retries permitted until 2026-07-29 04:04:21.867592728 +0000 UTC m=+42.726967432 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-registry-tls") pod "image-registry-55d9f6d798-l5rf7" (UID: "f194f797-9d02-465f-9293-e5e6d4226a21") : secret "image-registry-tls" not found Jul 29 04:04:17.868691 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:17.868653 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zjpnt" podStartSLOduration=9.919931481 podStartE2EDuration="38.868639271s" totalImagesPullingTime="28.94870779s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 04:03:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:04:17.868317109 +0000 UTC m=+38.727691826" watchObservedRunningTime="2026-07-29 04:04:17.868639271 +0000 UTC m=+38.728013994" Jul 29 04:04:18.069429 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:18.069225 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1deee91e-bb4a-4511-953d-9fd13697bdc1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-56kwm\" (UID: \"1deee91e-bb4a-4511-953d-9fd13697bdc1\") " pod="openshift-insights/insights-runtime-extractor-56kwm" Jul 29 04:04:18.069600 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:18.069457 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1deee91e-bb4a-4511-953d-9fd13697bdc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-56kwm\" (UID: \"1deee91e-bb4a-4511-953d-9fd13697bdc1\") " pod="openshift-insights/insights-runtime-extractor-56kwm" Jul 29 04:04:18.069648 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:18.069309 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 04:04:18.069713 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:18.069685 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1deee91e-bb4a-4511-953d-9fd13697bdc1-kube-rbac-proxy-cm podName:1deee91e-bb4a-4511-953d-9fd13697bdc1 nodeName:}" failed. No retries permitted until 2026-07-29 04:04:20.069671095 +0000 UTC m=+40.929045810 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1deee91e-bb4a-4511-953d-9fd13697bdc1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-56kwm" (UID: "1deee91e-bb4a-4511-953d-9fd13697bdc1") : configmap "kube-rbac-proxy" not found Jul 29 04:04:18.069964 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:18.069630 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 04:04:18.070046 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:18.069994 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1deee91e-bb4a-4511-953d-9fd13697bdc1-insights-runtime-extractor-tls podName:1deee91e-bb4a-4511-953d-9fd13697bdc1 nodeName:}" failed. No retries permitted until 2026-07-29 04:04:20.069982297 +0000 UTC m=+40.929357017 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1deee91e-bb4a-4511-953d-9fd13697bdc1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-56kwm" (UID: "1deee91e-bb4a-4511-953d-9fd13697bdc1") : secret "insights-runtime-extractor-tls" not found Jul 29 04:04:18.232941 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:18.232904 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-dxflj"] Jul 29 04:04:18.257005 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:18.256974 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-dxflj"] Jul 29 04:04:18.257167 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:18.257118 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-dxflj" Jul 29 04:04:18.259936 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:18.259911 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 29 04:04:18.260180 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:18.260160 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 29 04:04:18.260407 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:18.260391 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 29 04:04:18.260726 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:18.260705 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 29 04:04:18.260966 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:18.260945 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-c29b9\"" Jul 29 04:04:18.371295 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:18.371258 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2ae4f7ce-2b2a-4fdc-abb8-b353316bf714-signing-key\") pod \"service-ca-658d7b7c78-dxflj\" (UID: \"2ae4f7ce-2b2a-4fdc-abb8-b353316bf714\") " pod="openshift-service-ca/service-ca-658d7b7c78-dxflj" Jul 29 04:04:18.371459 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:18.371324 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2ae4f7ce-2b2a-4fdc-abb8-b353316bf714-signing-cabundle\") pod \"service-ca-658d7b7c78-dxflj\" (UID: \"2ae4f7ce-2b2a-4fdc-abb8-b353316bf714\") " pod="openshift-service-ca/service-ca-658d7b7c78-dxflj" Jul 29 04:04:18.371459 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:18.371387 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86f6q\" (UniqueName: \"kubernetes.io/projected/2ae4f7ce-2b2a-4fdc-abb8-b353316bf714-kube-api-access-86f6q\") pod \"service-ca-658d7b7c78-dxflj\" (UID: \"2ae4f7ce-2b2a-4fdc-abb8-b353316bf714\") " pod="openshift-service-ca/service-ca-658d7b7c78-dxflj" Jul 29 04:04:18.471992 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:18.471959 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2ae4f7ce-2b2a-4fdc-abb8-b353316bf714-signing-cabundle\") pod \"service-ca-658d7b7c78-dxflj\" (UID: \"2ae4f7ce-2b2a-4fdc-abb8-b353316bf714\") " pod="openshift-service-ca/service-ca-658d7b7c78-dxflj" Jul 29 04:04:18.472161 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:18.471999 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86f6q\" (UniqueName: \"kubernetes.io/projected/2ae4f7ce-2b2a-4fdc-abb8-b353316bf714-kube-api-access-86f6q\") pod \"service-ca-658d7b7c78-dxflj\" (UID: \"2ae4f7ce-2b2a-4fdc-abb8-b353316bf714\") " pod="openshift-service-ca/service-ca-658d7b7c78-dxflj" Jul 29 04:04:18.472161 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:18.472084 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2ae4f7ce-2b2a-4fdc-abb8-b353316bf714-signing-key\") pod \"service-ca-658d7b7c78-dxflj\" (UID: \"2ae4f7ce-2b2a-4fdc-abb8-b353316bf714\") " pod="openshift-service-ca/service-ca-658d7b7c78-dxflj" Jul 29 04:04:18.472714 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:18.472689 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2ae4f7ce-2b2a-4fdc-abb8-b353316bf714-signing-cabundle\") pod \"service-ca-658d7b7c78-dxflj\" (UID: \"2ae4f7ce-2b2a-4fdc-abb8-b353316bf714\") " pod="openshift-service-ca/service-ca-658d7b7c78-dxflj" Jul 29 04:04:18.475502 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:18.475471 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2ae4f7ce-2b2a-4fdc-abb8-b353316bf714-signing-key\") pod \"service-ca-658d7b7c78-dxflj\" (UID: \"2ae4f7ce-2b2a-4fdc-abb8-b353316bf714\") " pod="openshift-service-ca/service-ca-658d7b7c78-dxflj" Jul 29 04:04:18.480193 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:18.480169 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86f6q\" (UniqueName: \"kubernetes.io/projected/2ae4f7ce-2b2a-4fdc-abb8-b353316bf714-kube-api-access-86f6q\") pod \"service-ca-658d7b7c78-dxflj\" (UID: \"2ae4f7ce-2b2a-4fdc-abb8-b353316bf714\") " pod="openshift-service-ca/service-ca-658d7b7c78-dxflj" Jul 29 04:04:18.568298 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:18.568200 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-dxflj" Jul 29 04:04:18.849873 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:18.849799 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bprlw" event={"ID":"dcdad901-fa84-42eb-8e6d-70c86d2639cb","Type":"ContainerStarted","Data":"b4fb3f6a773178f74a8592ec6449757d820708a65215830bac6e27670517f090"} Jul 29 04:04:18.855728 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:18.855700 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-dxflj"] Jul 29 04:04:18.859462 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:04:18.859440 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae4f7ce_2b2a_4fdc_abb8_b353316bf714.slice/crio-1657abcab3aa0581a346b7a99d35868da05e560e881cf76f4956a237e81ee2e0 WatchSource:0}: Error finding container 1657abcab3aa0581a346b7a99d35868da05e560e881cf76f4956a237e81ee2e0: Status 404 returned error can't find the container with id 1657abcab3aa0581a346b7a99d35868da05e560e881cf76f4956a237e81ee2e0 Jul 29 04:04:18.869539 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:18.869494 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bprlw" podStartSLOduration=1.9601164469999999 podStartE2EDuration="4.869478385s" totalImagesPullingTime="2.909361938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:04:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:04:18.868163711 +0000 UTC m=+39.727538435" watchObservedRunningTime="2026-07-29 04:04:18.869478385 +0000 UTC m=+39.728853100" Jul 29 04:04:18.878433 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:18.878414 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v8d4n_dc47e5bf-dc39-4b81-ba67-5c00203489e0/dns-node-resolver/0.log" Jul 29 04:04:19.854325 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:19.854283 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-c8mlp" event={"ID":"2dafc4cf-b1a5-4907-8f24-a30ca4f08b30","Type":"ContainerStarted","Data":"417612bc89f47323a36f7113b18855efb9851784d188f5bec4130e89b33ffb63"} Jul 29 04:04:19.854806 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:19.854354 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-c8mlp" event={"ID":"2dafc4cf-b1a5-4907-8f24-a30ca4f08b30","Type":"ContainerStarted","Data":"b2419f76dfbccb61d73cae1e90601178c86da0a60b440e0ffc70ce8241122b9c"} Jul 29 04:04:19.855759 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:19.855728 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-dxflj" event={"ID":"2ae4f7ce-2b2a-4fdc-abb8-b353316bf714","Type":"ContainerStarted","Data":"1657abcab3aa0581a346b7a99d35868da05e560e881cf76f4956a237e81ee2e0"} Jul 29 04:04:19.871667 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:19.871608 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-c8mlp" podStartSLOduration=2.14142765 podStartE2EDuration="4.871589059s" totalImagesPullingTime="2.730161409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:04:19.870618126 +0000 UTC m=+40.729992850" watchObservedRunningTime="2026-07-29 04:04:19.871589059 +0000 UTC m=+40.730963784" Jul 29 04:04:19.877442 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:19.877416 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wwg6z_0055fae1-bd7e-4e87-834c-3a317313ae0c/node-ca/0.log" Jul 29 04:04:20.084693 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:20.084657 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1deee91e-bb4a-4511-953d-9fd13697bdc1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-56kwm\" (UID: \"1deee91e-bb4a-4511-953d-9fd13697bdc1\") " pod="openshift-insights/insights-runtime-extractor-56kwm" Jul 29 04:04:20.084845 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:20.084704 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1deee91e-bb4a-4511-953d-9fd13697bdc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-56kwm\" (UID: \"1deee91e-bb4a-4511-953d-9fd13697bdc1\") " pod="openshift-insights/insights-runtime-extractor-56kwm" Jul 29 04:04:20.084845 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:20.084775 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 04:04:20.084845 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:20.084807 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 04:04:20.084845 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:20.084838 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1deee91e-bb4a-4511-953d-9fd13697bdc1-kube-rbac-proxy-cm podName:1deee91e-bb4a-4511-953d-9fd13697bdc1 nodeName:}" failed. No retries permitted until 2026-07-29 04:04:24.084821849 +0000 UTC m=+44.944196577 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1deee91e-bb4a-4511-953d-9fd13697bdc1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-56kwm" (UID: "1deee91e-bb4a-4511-953d-9fd13697bdc1") : configmap "kube-rbac-proxy" not found Jul 29 04:04:20.084981 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:20.084852 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1deee91e-bb4a-4511-953d-9fd13697bdc1-insights-runtime-extractor-tls podName:1deee91e-bb4a-4511-953d-9fd13697bdc1 nodeName:}" failed. No retries permitted until 2026-07-29 04:04:24.084846215 +0000 UTC m=+44.944220917 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1deee91e-bb4a-4511-953d-9fd13697bdc1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-56kwm" (UID: "1deee91e-bb4a-4511-953d-9fd13697bdc1") : secret "insights-runtime-extractor-tls" not found Jul 29 04:04:20.477084 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:20.477048 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bprlw_dcdad901-fa84-42eb-8e6d-70c86d2639cb/serve-healthcheck-canary/0.log" Jul 29 04:04:20.678398 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:20.678364 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-c8mlp_2dafc4cf-b1a5-4907-8f24-a30ca4f08b30/migrator/0.log" Jul 29 04:04:20.880299 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:20.880230 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-c8mlp_2dafc4cf-b1a5-4907-8f24-a30ca4f08b30/graceful-termination/0.log" Jul 29 04:04:21.861549 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:21.861513 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-dxflj" event={"ID":"2ae4f7ce-2b2a-4fdc-abb8-b353316bf714","Type":"ContainerStarted","Data":"a171492e45ada1ea2f0bce02cabbc84cf814cb0f2c93950cb728368180782389"} Jul 29 04:04:21.876363 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:21.876310 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-dxflj" podStartSLOduration=1.5296684790000001 podStartE2EDuration="3.876289878s" totalImagesPullingTime="2.346621399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:04:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:04:21.875660655 +0000 UTC m=+42.735035380" watchObservedRunningTime="2026-07-29 04:04:21.876289878 +0000 UTC m=+42.735664606" Jul 29 04:04:21.898504 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:21.898476 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-registry-tls\") pod \"image-registry-55d9f6d798-l5rf7\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:21.898927 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:21.898610 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 04:04:21.898927 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:21.898634 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55d9f6d798-l5rf7: secret "image-registry-tls" not found Jul 29 04:04:21.898927 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:21.898699 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-registry-tls podName:f194f797-9d02-465f-9293-e5e6d4226a21 nodeName:}" failed. No retries permitted until 2026-07-29 04:04:29.898680952 +0000 UTC m=+50.758055678 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-registry-tls") pod "image-registry-55d9f6d798-l5rf7" (UID: "f194f797-9d02-465f-9293-e5e6d4226a21") : secret "image-registry-tls" not found Jul 29 04:04:21.898927 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:21.898724 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea98d97f-f580-434b-8d20-b687c0d951fd-metrics-tls\") pod \"dns-default-lfjbp\" (UID: \"ea98d97f-f580-434b-8d20-b687c0d951fd\") " pod="openshift-dns/dns-default-lfjbp" Jul 29 04:04:21.898927 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:21.898817 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 04:04:21.898927 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:21.898859 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea98d97f-f580-434b-8d20-b687c0d951fd-metrics-tls podName:ea98d97f-f580-434b-8d20-b687c0d951fd nodeName:}" failed. No retries permitted until 2026-07-29 04:04:29.898848067 +0000 UTC m=+50.758222793 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea98d97f-f580-434b-8d20-b687c0d951fd-metrics-tls") pod "dns-default-lfjbp" (UID: "ea98d97f-f580-434b-8d20-b687c0d951fd") : secret "dns-default-metrics-tls" not found Jul 29 04:04:24.117041 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:24.116996 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1deee91e-bb4a-4511-953d-9fd13697bdc1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-56kwm\" (UID: \"1deee91e-bb4a-4511-953d-9fd13697bdc1\") " pod="openshift-insights/insights-runtime-extractor-56kwm" Jul 29 04:04:24.117699 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:24.117058 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1deee91e-bb4a-4511-953d-9fd13697bdc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-56kwm\" (UID: \"1deee91e-bb4a-4511-953d-9fd13697bdc1\") " pod="openshift-insights/insights-runtime-extractor-56kwm" Jul 29 04:04:24.117699 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:24.117250 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 04:04:24.117699 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:04:24.117333 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1deee91e-bb4a-4511-953d-9fd13697bdc1-insights-runtime-extractor-tls podName:1deee91e-bb4a-4511-953d-9fd13697bdc1 nodeName:}" failed. No retries permitted until 2026-07-29 04:04:32.117317015 +0000 UTC m=+52.976691722 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1deee91e-bb4a-4511-953d-9fd13697bdc1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-56kwm" (UID: "1deee91e-bb4a-4511-953d-9fd13697bdc1") : secret "insights-runtime-extractor-tls" not found Jul 29 04:04:24.117699 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:24.117551 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1deee91e-bb4a-4511-953d-9fd13697bdc1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-56kwm\" (UID: \"1deee91e-bb4a-4511-953d-9fd13697bdc1\") " pod="openshift-insights/insights-runtime-extractor-56kwm" Jul 29 04:04:29.960664 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:29.960631 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea98d97f-f580-434b-8d20-b687c0d951fd-metrics-tls\") pod \"dns-default-lfjbp\" (UID: \"ea98d97f-f580-434b-8d20-b687c0d951fd\") " pod="openshift-dns/dns-default-lfjbp" Jul 29 04:04:29.961035 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:29.960711 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-registry-tls\") pod \"image-registry-55d9f6d798-l5rf7\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:29.963003 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:29.962973 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea98d97f-f580-434b-8d20-b687c0d951fd-metrics-tls\") pod \"dns-default-lfjbp\" (UID: \"ea98d97f-f580-434b-8d20-b687c0d951fd\") " pod="openshift-dns/dns-default-lfjbp" Jul 29 04:04:29.963233 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:29.963198 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-registry-tls\") pod \"image-registry-55d9f6d798-l5rf7\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:29.988186 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:29.987276 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lfjbp" Jul 29 04:04:29.999974 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:29.999945 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:30.134136 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:30.134098 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lfjbp"] Jul 29 04:04:30.137642 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:04:30.137614 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea98d97f_f580_434b_8d20_b687c0d951fd.slice/crio-4dcacec1185b875a4575f3c4f8fca215fa2d50679188ba483846113d8f333900 WatchSource:0}: Error finding container 4dcacec1185b875a4575f3c4f8fca215fa2d50679188ba483846113d8f333900: Status 404 returned error can't find the container with id 4dcacec1185b875a4575f3c4f8fca215fa2d50679188ba483846113d8f333900 Jul 29 04:04:30.152379 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:30.152354 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55d9f6d798-l5rf7"] Jul 29 04:04:30.154859 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:04:30.154834 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf194f797_9d02_465f_9293_e5e6d4226a21.slice/crio-f5ca3c0c251514061be10f0533823367709b3d2b1b5b7b92c0a4ab95557f40d2 WatchSource:0}: Error finding container f5ca3c0c251514061be10f0533823367709b3d2b1b5b7b92c0a4ab95557f40d2: Status 404 returned error can't find the container with id f5ca3c0c251514061be10f0533823367709b3d2b1b5b7b92c0a4ab95557f40d2 Jul 29 04:04:30.886257 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:30.886189 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" event={"ID":"f194f797-9d02-465f-9293-e5e6d4226a21","Type":"ContainerStarted","Data":"bd68d37a6efb92deaf529364c904104bf5d023cfb7f048b77fbd7775aa3d3544"} Jul 29 04:04:30.886455 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:30.886271 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" event={"ID":"f194f797-9d02-465f-9293-e5e6d4226a21","Type":"ContainerStarted","Data":"f5ca3c0c251514061be10f0533823367709b3d2b1b5b7b92c0a4ab95557f40d2"} Jul 29 04:04:30.886455 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:30.886316 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:04:30.887422 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:30.887397 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lfjbp" event={"ID":"ea98d97f-f580-434b-8d20-b687c0d951fd","Type":"ContainerStarted","Data":"4dcacec1185b875a4575f3c4f8fca215fa2d50679188ba483846113d8f333900"} Jul 29 04:04:30.906198 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:30.906149 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" podStartSLOduration=27.906134193 podStartE2EDuration="27.906134193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:04:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:04:30.905274388 +0000 UTC m=+51.764649112" watchObservedRunningTime="2026-07-29 04:04:30.906134193 +0000 UTC m=+51.765508917" Jul 29 04:04:32.180902 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:32.180860 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1deee91e-bb4a-4511-953d-9fd13697bdc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-56kwm\" (UID: \"1deee91e-bb4a-4511-953d-9fd13697bdc1\") " pod="openshift-insights/insights-runtime-extractor-56kwm" Jul 29 04:04:32.183518 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:32.183464 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1deee91e-bb4a-4511-953d-9fd13697bdc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-56kwm\" (UID: \"1deee91e-bb4a-4511-953d-9fd13697bdc1\") " pod="openshift-insights/insights-runtime-extractor-56kwm" Jul 29 04:04:32.220673 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:32.220632 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-56kwm" Jul 29 04:04:32.725574 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:32.725546 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-56kwm"] Jul 29 04:04:32.727771 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:04:32.727746 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1deee91e_bb4a_4511_953d_9fd13697bdc1.slice/crio-d0ebdb3a6bb77b8db6d4baef1e5ca7e7f0def20539f721ae6be85d9f11decf8d WatchSource:0}: Error finding container d0ebdb3a6bb77b8db6d4baef1e5ca7e7f0def20539f721ae6be85d9f11decf8d: Status 404 returned error can't find the container with id d0ebdb3a6bb77b8db6d4baef1e5ca7e7f0def20539f721ae6be85d9f11decf8d Jul 29 04:04:32.892895 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:32.892818 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-56kwm" event={"ID":"1deee91e-bb4a-4511-953d-9fd13697bdc1","Type":"ContainerStarted","Data":"d0ebdb3a6bb77b8db6d4baef1e5ca7e7f0def20539f721ae6be85d9f11decf8d"} Jul 29 04:04:33.903136 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:33.902354 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lfjbp" event={"ID":"ea98d97f-f580-434b-8d20-b687c0d951fd","Type":"ContainerStarted","Data":"4f3f4305afe3ae252730a968a4735b91629831bc2250011ff664d4046ee1230f"} Jul 29 04:04:33.903136 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:33.902459 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lfjbp" event={"ID":"ea98d97f-f580-434b-8d20-b687c0d951fd","Type":"ContainerStarted","Data":"5f7c99b4d0752f3e958c63e7acdc2cad07cd7c48e3784aad6c12db2549ddca51"} Jul 29 04:04:33.903136 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:33.902915 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lfjbp" Jul 29 04:04:33.918248 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:33.918175 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lfjbp" podStartSLOduration=17.088279006 podStartE2EDuration="19.918152453s" totalImagesPullingTime="2.829873447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:04:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:04:33.917535819 +0000 UTC m=+54.776910555" watchObservedRunningTime="2026-07-29 04:04:33.918152453 +0000 UTC m=+54.777527180" Jul 29 04:04:37.917082 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:37.917045 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-56kwm" event={"ID":"1deee91e-bb4a-4511-953d-9fd13697bdc1","Type":"ContainerStarted","Data":"b9df5143ed07025b1d55b41a93168964195680219cf9bff2882450e3e3ff1dbb"} Jul 29 04:04:39.822382 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:39.822351 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7792" Jul 29 04:04:40.060371 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.059761 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7b9b86fd-tst67"] Jul 29 04:04:40.160525 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.160442 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56447948cd-tvw7p"] Jul 29 04:04:40.160694 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.160549 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7b9b86fd-tst67" Jul 29 04:04:40.165431 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.163747 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 29 04:04:40.165431 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.164020 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 29 04:04:40.165431 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.164294 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 29 04:04:40.165431 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.164536 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 29 04:04:40.191250 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.191221 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-589f57d8bf-fvj8z"] Jul 29 04:04:40.191456 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.191351 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56447948cd-tvw7p" Jul 29 04:04:40.194135 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.194107 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-bhjnx\"" Jul 29 04:04:40.194295 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.194190 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 29 04:04:40.218302 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.218271 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7b9b86fd-tst67"] Jul 29 04:04:40.218467 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.218324 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55d9f6d798-l5rf7"] Jul 29 04:04:40.218467 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.218355 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56447948cd-tvw7p"] Jul 29 04:04:40.218467 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.218371 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-589f57d8bf-fvj8z"] Jul 29 04:04:40.218467 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.218391 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-bbfb64c88-lv4w2"] Jul 29 04:04:40.218663 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.218478 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-589f57d8bf-fvj8z" Jul 29 04:04:40.221872 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.221852 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 29 04:04:40.221992 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.221927 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 29 04:04:40.222063 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.222054 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 29 04:04:40.222529 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.222510 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 29 04:04:40.244384 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.244358 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bbfb64c88-lv4w2"] Jul 29 04:04:40.244610 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.244500 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bbfb64c88-lv4w2" Jul 29 04:04:40.244610 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.244574 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d0b6a0c-2568-41c7-bef8-a32dee9ca796-tmp\") pod \"klusterlet-addon-workmgr-7b7b9b86fd-tst67\" (UID: \"8d0b6a0c-2568-41c7-bef8-a32dee9ca796\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7b9b86fd-tst67" Jul 29 04:04:40.244610 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.244605 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7eccdff2-3f26-47bd-b2b9-c7e90dffae96-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-56447948cd-tvw7p\" (UID: \"7eccdff2-3f26-47bd-b2b9-c7e90dffae96\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56447948cd-tvw7p" Jul 29 04:04:40.244774 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.244636 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8d0b6a0c-2568-41c7-bef8-a32dee9ca796-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b7b9b86fd-tst67\" (UID: \"8d0b6a0c-2568-41c7-bef8-a32dee9ca796\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7b9b86fd-tst67" Jul 29 04:04:40.244774 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.244686 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-484dz\" (UniqueName: \"kubernetes.io/projected/7eccdff2-3f26-47bd-b2b9-c7e90dffae96-kube-api-access-484dz\") pod \"managed-serviceaccount-addon-agent-56447948cd-tvw7p\" (UID: \"7eccdff2-3f26-47bd-b2b9-c7e90dffae96\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56447948cd-tvw7p" Jul 29 04:04:40.244774 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.244724 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zl8d\" (UniqueName: \"kubernetes.io/projected/8d0b6a0c-2568-41c7-bef8-a32dee9ca796-kube-api-access-6zl8d\") pod \"klusterlet-addon-workmgr-7b7b9b86fd-tst67\" (UID: \"8d0b6a0c-2568-41c7-bef8-a32dee9ca796\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7b9b86fd-tst67" Jul 29 04:04:40.345143 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.345114 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f243811-a083-4a5d-9043-47b372431bce-trusted-ca\") pod \"image-registry-bbfb64c88-lv4w2\" (UID: \"3f243811-a083-4a5d-9043-47b372431bce\") " pod="openshift-image-registry/image-registry-bbfb64c88-lv4w2" Jul 29 04:04:40.345352 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.345158 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f243811-a083-4a5d-9043-47b372431bce-bound-sa-token\") pod \"image-registry-bbfb64c88-lv4w2\" (UID: \"3f243811-a083-4a5d-9043-47b372431bce\") " pod="openshift-image-registry/image-registry-bbfb64c88-lv4w2" Jul 29 04:04:40.345352 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.345181 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zl8d\" (UniqueName: \"kubernetes.io/projected/8d0b6a0c-2568-41c7-bef8-a32dee9ca796-kube-api-access-6zl8d\") pod \"klusterlet-addon-workmgr-7b7b9b86fd-tst67\" (UID: \"8d0b6a0c-2568-41c7-bef8-a32dee9ca796\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7b9b86fd-tst67" Jul 29 04:04:40.345352 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.345239 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9ccc665c-b410-4af4-ac83-e9ca0d632bcf-hub\") pod \"cluster-proxy-proxy-agent-589f57d8bf-fvj8z\" (UID: \"9ccc665c-b410-4af4-ac83-e9ca0d632bcf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-589f57d8bf-fvj8z" Jul 29 04:04:40.345352 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.345271 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9ccc665c-b410-4af4-ac83-e9ca0d632bcf-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-589f57d8bf-fvj8z\" (UID: \"9ccc665c-b410-4af4-ac83-e9ca0d632bcf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-589f57d8bf-fvj8z" Jul 29 04:04:40.345352 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.345291 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f243811-a083-4a5d-9043-47b372431bce-image-registry-private-configuration\") pod \"image-registry-bbfb64c88-lv4w2\" (UID: \"3f243811-a083-4a5d-9043-47b372431bce\") " pod="openshift-image-registry/image-registry-bbfb64c88-lv4w2" Jul 29 04:04:40.345352 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.345319 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ktfk\" (UniqueName: \"kubernetes.io/projected/3f243811-a083-4a5d-9043-47b372431bce-kube-api-access-9ktfk\") pod \"image-registry-bbfb64c88-lv4w2\" (UID: \"3f243811-a083-4a5d-9043-47b372431bce\") " pod="openshift-image-registry/image-registry-bbfb64c88-lv4w2" Jul 29 04:04:40.345352 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.345341 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f243811-a083-4a5d-9043-47b372431bce-registry-certificates\") pod \"image-registry-bbfb64c88-lv4w2\" (UID: \"3f243811-a083-4a5d-9043-47b372431bce\") " pod="openshift-image-registry/image-registry-bbfb64c88-lv4w2" Jul 29 04:04:40.345708 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.345387 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d0b6a0c-2568-41c7-bef8-a32dee9ca796-tmp\") pod \"klusterlet-addon-workmgr-7b7b9b86fd-tst67\" (UID: \"8d0b6a0c-2568-41c7-bef8-a32dee9ca796\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7b9b86fd-tst67" Jul 29 04:04:40.345708 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.345435 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ccc665c-b410-4af4-ac83-e9ca0d632bcf-additional-service-ca\") pod \"cluster-proxy-proxy-agent-589f57d8bf-fvj8z\" (UID: \"9ccc665c-b410-4af4-ac83-e9ca0d632bcf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-589f57d8bf-fvj8z" Jul 29 04:04:40.345708 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.345476 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7eccdff2-3f26-47bd-b2b9-c7e90dffae96-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-56447948cd-tvw7p\" (UID: \"7eccdff2-3f26-47bd-b2b9-c7e90dffae96\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56447948cd-tvw7p" Jul 29 04:04:40.345708 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.345522 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8d0b6a0c-2568-41c7-bef8-a32dee9ca796-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b7b9b86fd-tst67\" (UID: \"8d0b6a0c-2568-41c7-bef8-a32dee9ca796\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7b9b86fd-tst67" Jul 29 04:04:40.345708 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.345554 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f243811-a083-4a5d-9043-47b372431bce-registry-tls\") pod \"image-registry-bbfb64c88-lv4w2\" (UID: \"3f243811-a083-4a5d-9043-47b372431bce\") " pod="openshift-image-registry/image-registry-bbfb64c88-lv4w2" Jul 29 04:04:40.345708 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.345581 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f243811-a083-4a5d-9043-47b372431bce-ca-trust-extracted\") pod \"image-registry-bbfb64c88-lv4w2\" (UID: \"3f243811-a083-4a5d-9043-47b372431bce\") " pod="openshift-image-registry/image-registry-bbfb64c88-lv4w2" Jul 29 04:04:40.345708 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.345610 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9ccc665c-b410-4af4-ac83-e9ca0d632bcf-ca\") pod \"cluster-proxy-proxy-agent-589f57d8bf-fvj8z\" (UID: \"9ccc665c-b410-4af4-ac83-e9ca0d632bcf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-589f57d8bf-fvj8z" Jul 29 04:04:40.345708 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.345662 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttwk5\" (UniqueName: \"kubernetes.io/projected/9ccc665c-b410-4af4-ac83-e9ca0d632bcf-kube-api-access-ttwk5\") pod \"cluster-proxy-proxy-agent-589f57d8bf-fvj8z\" (UID: \"9ccc665c-b410-4af4-ac83-e9ca0d632bcf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-589f57d8bf-fvj8z" Jul 29 04:04:40.345708 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.345703 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d0b6a0c-2568-41c7-bef8-a32dee9ca796-tmp\") pod \"klusterlet-addon-workmgr-7b7b9b86fd-tst67\" (UID: \"8d0b6a0c-2568-41c7-bef8-a32dee9ca796\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7b9b86fd-tst67" Jul 29 04:04:40.346030 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.345774 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-484dz\" (UniqueName: \"kubernetes.io/projected/7eccdff2-3f26-47bd-b2b9-c7e90dffae96-kube-api-access-484dz\") pod \"managed-serviceaccount-addon-agent-56447948cd-tvw7p\" (UID: \"7eccdff2-3f26-47bd-b2b9-c7e90dffae96\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56447948cd-tvw7p" Jul 29 04:04:40.346030 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.345808 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9ccc665c-b410-4af4-ac83-e9ca0d632bcf-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-589f57d8bf-fvj8z\" (UID: \"9ccc665c-b410-4af4-ac83-e9ca0d632bcf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-589f57d8bf-fvj8z" Jul 29 04:04:40.346030 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.345844 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f243811-a083-4a5d-9043-47b372431bce-installation-pull-secrets\") pod \"image-registry-bbfb64c88-lv4w2\" (UID: \"3f243811-a083-4a5d-9043-47b372431bce\") " pod="openshift-image-registry/image-registry-bbfb64c88-lv4w2" Jul 29 04:04:40.348016 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.347996 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8d0b6a0c-2568-41c7-bef8-a32dee9ca796-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b7b9b86fd-tst67\" (UID: \"8d0b6a0c-2568-41c7-bef8-a32dee9ca796\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7b9b86fd-tst67" Jul 29 04:04:40.348108 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.347997 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7eccdff2-3f26-47bd-b2b9-c7e90dffae96-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-56447948cd-tvw7p\" (UID: \"7eccdff2-3f26-47bd-b2b9-c7e90dffae96\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56447948cd-tvw7p" Jul 29 04:04:40.354686 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.354661 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-484dz\" (UniqueName: \"kubernetes.io/projected/7eccdff2-3f26-47bd-b2b9-c7e90dffae96-kube-api-access-484dz\") pod \"managed-serviceaccount-addon-agent-56447948cd-tvw7p\" (UID: \"7eccdff2-3f26-47bd-b2b9-c7e90dffae96\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56447948cd-tvw7p" Jul 29 04:04:40.354864 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.354842 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zl8d\" (UniqueName: \"kubernetes.io/projected/8d0b6a0c-2568-41c7-bef8-a32dee9ca796-kube-api-access-6zl8d\") pod \"klusterlet-addon-workmgr-7b7b9b86fd-tst67\" (UID: \"8d0b6a0c-2568-41c7-bef8-a32dee9ca796\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7b9b86fd-tst67" Jul 29 04:04:40.446970 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.446928 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f243811-a083-4a5d-9043-47b372431bce-registry-certificates\") pod \"image-registry-bbfb64c88-lv4w2\" (UID: \"3f243811-a083-4a5d-9043-47b372431bce\") " pod="openshift-image-registry/image-registry-bbfb64c88-lv4w2" Jul 29 04:04:40.447133 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.446989 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ccc665c-b410-4af4-ac83-e9ca0d632bcf-additional-service-ca\") pod \"cluster-proxy-proxy-agent-589f57d8bf-fvj8z\" (UID: \"9ccc665c-b410-4af4-ac83-e9ca0d632bcf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-589f57d8bf-fvj8z" Jul 29 04:04:40.447133 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.447011 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f243811-a083-4a5d-9043-47b372431bce-registry-tls\") pod \"image-registry-bbfb64c88-lv4w2\" (UID: \"3f243811-a083-4a5d-9043-47b372431bce\") " pod="openshift-image-registry/image-registry-bbfb64c88-lv4w2" Jul 29 04:04:40.447133 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.447028 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f243811-a083-4a5d-9043-47b372431bce-ca-trust-extracted\") pod \"image-registry-bbfb64c88-lv4w2\" (UID: \"3f243811-a083-4a5d-9043-47b372431bce\") " pod="openshift-image-registry/image-registry-bbfb64c88-lv4w2" Jul 29 04:04:40.447133 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.447045 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9ccc665c-b410-4af4-ac83-e9ca0d632bcf-ca\") pod \"cluster-proxy-proxy-agent-589f57d8bf-fvj8z\" (UID: \"9ccc665c-b410-4af4-ac83-e9ca0d632bcf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-589f57d8bf-fvj8z" Jul 29 04:04:40.447133 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.447063 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttwk5\" (UniqueName: \"kubernetes.io/projected/9ccc665c-b410-4af4-ac83-e9ca0d632bcf-kube-api-access-ttwk5\") pod \"cluster-proxy-proxy-agent-589f57d8bf-fvj8z\" (UID: \"9ccc665c-b410-4af4-ac83-e9ca0d632bcf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-589f57d8bf-fvj8z" Jul 29 04:04:40.447133 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.447099 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9ccc665c-b410-4af4-ac83-e9ca0d632bcf-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-589f57d8bf-fvj8z\" (UID: \"9ccc665c-b410-4af4-ac83-e9ca0d632bcf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-589f57d8bf-fvj8z" Jul 29 04:04:40.447547 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.447508 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f243811-a083-4a5d-9043-47b372431bce-ca-trust-extracted\") pod \"image-registry-bbfb64c88-lv4w2\" (UID: \"3f243811-a083-4a5d-9043-47b372431bce\") " pod="openshift-image-registry/image-registry-bbfb64c88-lv4w2" Jul 29 04:04:40.447547 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.447509 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f243811-a083-4a5d-9043-47b372431bce-installation-pull-secrets\") pod \"image-registry-bbfb64c88-lv4w2\" (UID: \"3f243811-a083-4a5d-9043-47b372431bce\") " pod="openshift-image-registry/image-registry-bbfb64c88-lv4w2" Jul 29 04:04:40.447702 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.447590 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f243811-a083-4a5d-9043-47b372431bce-trusted-ca\") pod \"image-registry-bbfb64c88-lv4w2\" (UID: \"3f243811-a083-4a5d-9043-47b372431bce\") " pod="openshift-image-registry/image-registry-bbfb64c88-lv4w2" Jul 29 04:04:40.447702 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.447644 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f243811-a083-4a5d-9043-47b372431bce-bound-sa-token\") pod \"image-registry-bbfb64c88-lv4w2\" (UID: \"3f243811-a083-4a5d-9043-47b372431bce\") " pod="openshift-image-registry/image-registry-bbfb64c88-lv4w2" Jul 29 04:04:40.447702 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.447679 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9ccc665c-b410-4af4-ac83-e9ca0d632bcf-hub\") pod \"cluster-proxy-proxy-agent-589f57d8bf-fvj8z\" (UID: \"9ccc665c-b410-4af4-ac83-e9ca0d632bcf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-589f57d8bf-fvj8z" Jul 29 04:04:40.447852 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.447706 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9ccc665c-b410-4af4-ac83-e9ca0d632bcf-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-589f57d8bf-fvj8z\" (UID: \"9ccc665c-b410-4af4-ac83-e9ca0d632bcf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-589f57d8bf-fvj8z" Jul 29 04:04:40.447852 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.447732 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f243811-a083-4a5d-9043-47b372431bce-image-registry-private-configuration\") pod \"image-registry-bbfb64c88-lv4w2\" (UID: \"3f243811-a083-4a5d-9043-47b372431bce\") " pod="openshift-image-registry/image-registry-bbfb64c88-lv4w2" Jul 29 04:04:40.447852 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.447763 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ktfk\" (UniqueName: \"kubernetes.io/projected/3f243811-a083-4a5d-9043-47b372431bce-kube-api-access-9ktfk\") pod \"image-registry-bbfb64c88-lv4w2\" (UID: \"3f243811-a083-4a5d-9043-47b372431bce\") " pod="openshift-image-registry/image-registry-bbfb64c88-lv4w2" Jul 29 04:04:40.447852 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.447816 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ccc665c-b410-4af4-ac83-e9ca0d632bcf-additional-service-ca\") pod \"cluster-proxy-proxy-agent-589f57d8bf-fvj8z\" (UID: \"9ccc665c-b410-4af4-ac83-e9ca0d632bcf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-589f57d8bf-fvj8z" Jul 29 04:04:40.447852 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.447825 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f243811-a083-4a5d-9043-47b372431bce-registry-certificates\") pod \"image-registry-bbfb64c88-lv4w2\" (UID: \"3f243811-a083-4a5d-9043-47b372431bce\") " pod="openshift-image-registry/image-registry-bbfb64c88-lv4w2" Jul 29 04:04:40.448838 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.448812 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f243811-a083-4a5d-9043-47b372431bce-trusted-ca\") pod \"image-registry-bbfb64c88-lv4w2\" (UID: \"3f243811-a083-4a5d-9043-47b372431bce\") " pod="openshift-image-registry/image-registry-bbfb64c88-lv4w2" Jul 29 04:04:40.450426 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.450404 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f243811-a083-4a5d-9043-47b372431bce-registry-tls\") pod \"image-registry-bbfb64c88-lv4w2\" (UID: \"3f243811-a083-4a5d-9043-47b372431bce\") " pod="openshift-image-registry/image-registry-bbfb64c88-lv4w2" Jul 29 04:04:40.450532 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.450444 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f243811-a083-4a5d-9043-47b372431bce-installation-pull-secrets\") pod \"image-registry-bbfb64c88-lv4w2\" (UID: \"3f243811-a083-4a5d-9043-47b372431bce\") " pod="openshift-image-registry/image-registry-bbfb64c88-lv4w2" Jul 29 04:04:40.450532 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.450502 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9ccc665c-b410-4af4-ac83-e9ca0d632bcf-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-589f57d8bf-fvj8z\" (UID: \"9ccc665c-b410-4af4-ac83-e9ca0d632bcf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-589f57d8bf-fvj8z" Jul 29 04:04:40.450612 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.450581 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9ccc665c-b410-4af4-ac83-e9ca0d632bcf-ca\") pod \"cluster-proxy-proxy-agent-589f57d8bf-fvj8z\" (UID: \"9ccc665c-b410-4af4-ac83-e9ca0d632bcf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-589f57d8bf-fvj8z" Jul 29 04:04:40.451030 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.451007 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f243811-a083-4a5d-9043-47b372431bce-image-registry-private-configuration\") pod \"image-registry-bbfb64c88-lv4w2\" (UID: \"3f243811-a083-4a5d-9043-47b372431bce\") " pod="openshift-image-registry/image-registry-bbfb64c88-lv4w2" Jul 29 04:04:40.451126 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.451088 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9ccc665c-b410-4af4-ac83-e9ca0d632bcf-hub\") pod \"cluster-proxy-proxy-agent-589f57d8bf-fvj8z\" (UID: \"9ccc665c-b410-4af4-ac83-e9ca0d632bcf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-589f57d8bf-fvj8z" Jul 29 04:04:40.451307 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.451291 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9ccc665c-b410-4af4-ac83-e9ca0d632bcf-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-589f57d8bf-fvj8z\" (UID: \"9ccc665c-b410-4af4-ac83-e9ca0d632bcf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-589f57d8bf-fvj8z" Jul 29 04:04:40.455520 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.455496 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ktfk\" (UniqueName: \"kubernetes.io/projected/3f243811-a083-4a5d-9043-47b372431bce-kube-api-access-9ktfk\") pod \"image-registry-bbfb64c88-lv4w2\" (UID: \"3f243811-a083-4a5d-9043-47b372431bce\") " pod="openshift-image-registry/image-registry-bbfb64c88-lv4w2" Jul 29 04:04:40.455759 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.455742 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f243811-a083-4a5d-9043-47b372431bce-bound-sa-token\") pod \"image-registry-bbfb64c88-lv4w2\" (UID: \"3f243811-a083-4a5d-9043-47b372431bce\") " pod="openshift-image-registry/image-registry-bbfb64c88-lv4w2" Jul 29 04:04:40.456223 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.456189 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttwk5\" (UniqueName: \"kubernetes.io/projected/9ccc665c-b410-4af4-ac83-e9ca0d632bcf-kube-api-access-ttwk5\") pod \"cluster-proxy-proxy-agent-589f57d8bf-fvj8z\" (UID: \"9ccc665c-b410-4af4-ac83-e9ca0d632bcf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-589f57d8bf-fvj8z" Jul 29 04:04:40.471845 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.471824 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7b9b86fd-tst67" Jul 29 04:04:40.513441 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.513412 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56447948cd-tvw7p" Jul 29 04:04:40.529600 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.529577 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-589f57d8bf-fvj8z" Jul 29 04:04:40.562553 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:40.562344 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bbfb64c88-lv4w2" Jul 29 04:04:41.354647 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:41.354615 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7b9b86fd-tst67"] Jul 29 04:04:41.358312 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:04:41.358283 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d0b6a0c_2568_41c7_bef8_a32dee9ca796.slice/crio-352f9f27b952b891f9a302daacc0918e26920983568a313494f451003315aa9a WatchSource:0}: Error finding container 352f9f27b952b891f9a302daacc0918e26920983568a313494f451003315aa9a: Status 404 returned error can't find the container with id 352f9f27b952b891f9a302daacc0918e26920983568a313494f451003315aa9a Jul 29 04:04:41.577387 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:41.577284 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56447948cd-tvw7p"] Jul 29 04:04:41.580222 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:04:41.580180 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7eccdff2_3f26_47bd_b2b9_c7e90dffae96.slice/crio-45e5501eb5b96ca564e7c23084a9dc2df7889a2353a2a5fb5440371015ea1a3b WatchSource:0}: Error finding container 45e5501eb5b96ca564e7c23084a9dc2df7889a2353a2a5fb5440371015ea1a3b: Status 404 returned error can't find the container with id 45e5501eb5b96ca564e7c23084a9dc2df7889a2353a2a5fb5440371015ea1a3b Jul 29 04:04:41.584416 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:41.584392 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bbfb64c88-lv4w2"] Jul 29 04:04:41.587309 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:04:41.587279 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f243811_a083_4a5d_9043_47b372431bce.slice/crio-4b8f7452ad77f36e5421a2b900116427576652a7826fcb3451e551db3838bede WatchSource:0}: Error finding container 4b8f7452ad77f36e5421a2b900116427576652a7826fcb3451e551db3838bede: Status 404 returned error can't find the container with id 4b8f7452ad77f36e5421a2b900116427576652a7826fcb3451e551db3838bede Jul 29 04:04:41.590268 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:41.590245 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-589f57d8bf-fvj8z"] Jul 29 04:04:41.593503 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:04:41.593481 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ccc665c_b410_4af4_ac83_e9ca0d632bcf.slice/crio-dc480185691282cc6df950c110ee13bd985df48ada37f79fa943327b8a0d6222 WatchSource:0}: Error finding container dc480185691282cc6df950c110ee13bd985df48ada37f79fa943327b8a0d6222: Status 404 returned error can't find the container with id dc480185691282cc6df950c110ee13bd985df48ada37f79fa943327b8a0d6222 Jul 29 04:04:41.933165 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:41.933021 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bbfb64c88-lv4w2" event={"ID":"3f243811-a083-4a5d-9043-47b372431bce","Type":"ContainerStarted","Data":"a2304bd59a1a29aa25dccf5fc49ac69b70bb9d3cf9b664dfb139682392195dae"} Jul 29 04:04:41.933165 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:41.933085 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bbfb64c88-lv4w2" event={"ID":"3f243811-a083-4a5d-9043-47b372431bce","Type":"ContainerStarted","Data":"4b8f7452ad77f36e5421a2b900116427576652a7826fcb3451e551db3838bede"} Jul 29 04:04:41.933165 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:41.933115 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-bbfb64c88-lv4w2" Jul 29 04:04:41.934619 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:41.934583 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56447948cd-tvw7p" event={"ID":"7eccdff2-3f26-47bd-b2b9-c7e90dffae96","Type":"ContainerStarted","Data":"45e5501eb5b96ca564e7c23084a9dc2df7889a2353a2a5fb5440371015ea1a3b"} Jul 29 04:04:41.935742 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:41.935708 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-589f57d8bf-fvj8z" event={"ID":"9ccc665c-b410-4af4-ac83-e9ca0d632bcf","Type":"ContainerStarted","Data":"dc480185691282cc6df950c110ee13bd985df48ada37f79fa943327b8a0d6222"} Jul 29 04:04:41.936906 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:41.936878 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7b9b86fd-tst67" event={"ID":"8d0b6a0c-2568-41c7-bef8-a32dee9ca796","Type":"ContainerStarted","Data":"352f9f27b952b891f9a302daacc0918e26920983568a313494f451003315aa9a"} Jul 29 04:04:41.938635 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:41.938615 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-56kwm" event={"ID":"1deee91e-bb4a-4511-953d-9fd13697bdc1","Type":"ContainerStarted","Data":"f1021641a8f4fd48b10e49a6b44baa64e5730a0322b5e23e7a7428474555c872"} Jul 29 04:04:41.950897 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:41.950856 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-bbfb64c88-lv4w2" podStartSLOduration=1.950843264 podStartE2EDuration="1.950843264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:04:41.950543044 +0000 UTC m=+62.809917769" watchObservedRunningTime="2026-07-29 04:04:41.950843264 +0000 UTC m=+62.810217977" Jul 29 04:04:43.909591 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:43.909489 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lfjbp" Jul 29 04:04:45.298818 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:45.298781 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fa6e244-be92-4245-a54e-7aa303802f4b-metrics-certs\") pod \"network-metrics-daemon-zbrk8\" (UID: \"9fa6e244-be92-4245-a54e-7aa303802f4b\") " pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:04:45.301151 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:45.301126 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 04:04:45.312058 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:45.312010 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fa6e244-be92-4245-a54e-7aa303802f4b-metrics-certs\") pod \"network-metrics-daemon-zbrk8\" (UID: \"9fa6e244-be92-4245-a54e-7aa303802f4b\") " pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:04:45.399679 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:45.399635 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b-original-pull-secret\") pod \"global-pull-secret-syncer-b9zpd\" (UID: \"3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b\") " pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:04:45.402198 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:45.402174 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 04:04:45.412271 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:45.412242 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b-original-pull-secret\") pod \"global-pull-secret-syncer-b9zpd\" (UID: \"3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b\") " pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:04:45.500788 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:45.500746 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lqmq\" (UniqueName: \"kubernetes.io/projected/350381e6-34a2-4f35-90d4-460ed24c7096-kube-api-access-7lqmq\") pod \"network-check-target-x6vhn\" (UID: \"350381e6-34a2-4f35-90d4-460ed24c7096\") " pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:04:45.503248 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:45.503222 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 04:04:45.513896 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:45.513872 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 04:04:45.525227 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:45.525168 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lqmq\" (UniqueName: \"kubernetes.io/projected/350381e6-34a2-4f35-90d4-460ed24c7096-kube-api-access-7lqmq\") pod \"network-check-target-x6vhn\" (UID: \"350381e6-34a2-4f35-90d4-460ed24c7096\") " pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:04:45.597391 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:45.597317 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bzqwr\"" Jul 29 04:04:45.601155 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:45.601127 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9zpd" Jul 29 04:04:45.605217 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:45.605183 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:04:45.607068 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:45.607048 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mrbfg\"" Jul 29 04:04:45.615860 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:45.615840 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbrk8" Jul 29 04:04:50.230920 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:50.230368 2579 patch_prober.go:28] interesting pod/image-registry-55d9f6d798-l5rf7 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 29 04:04:50.230920 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:50.230446 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" podUID="f194f797-9d02-465f-9293-e5e6d4226a21" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 04:04:50.322447 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:50.322403 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zbrk8"] Jul 29 04:04:50.344756 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:50.342678 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b9zpd"] Jul 29 04:04:50.344756 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:50.342785 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x6vhn"] Jul 29 04:04:50.348929 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:04:50.348902 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod350381e6_34a2_4f35_90d4_460ed24c7096.slice/crio-fd7b0af714d311022c7228be62764c117ad7c623d8f99d281179fa60bc524612 WatchSource:0}: Error finding container fd7b0af714d311022c7228be62764c117ad7c623d8f99d281179fa60bc524612: Status 404 returned error can't find the container with id fd7b0af714d311022c7228be62764c117ad7c623d8f99d281179fa60bc524612 Jul 29 04:04:50.990277 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:50.990235 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x6vhn" event={"ID":"350381e6-34a2-4f35-90d4-460ed24c7096","Type":"ContainerStarted","Data":"fd7b0af714d311022c7228be62764c117ad7c623d8f99d281179fa60bc524612"} Jul 29 04:04:50.992042 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:50.992007 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zbrk8" event={"ID":"9fa6e244-be92-4245-a54e-7aa303802f4b","Type":"ContainerStarted","Data":"c662f927487253945480842387fe770e9454ccaca8dbf63f336683b4a185062e"} Jul 29 04:04:50.993851 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:50.993820 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56447948cd-tvw7p" event={"ID":"7eccdff2-3f26-47bd-b2b9-c7e90dffae96","Type":"ContainerStarted","Data":"e0b35ccb346520f5200917f3c918f79a89d263dee80b6136329bd03e8aff6c17"} Jul 29 04:04:50.997554 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:50.997523 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-589f57d8bf-fvj8z" event={"ID":"9ccc665c-b410-4af4-ac83-e9ca0d632bcf","Type":"ContainerStarted","Data":"c34b86c98d34aa2cf8b7f1bb4d8a67c96c42c6337de29c2e9f7656338cf5d114"} Jul 29 04:04:50.997674 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:50.997566 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-589f57d8bf-fvj8z" event={"ID":"9ccc665c-b410-4af4-ac83-e9ca0d632bcf","Type":"ContainerStarted","Data":"25780b5884e474e3f18ec981813cce097f27c5b473e68d4806185655b1414c26"} Jul 29 04:04:50.997674 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:50.997585 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-589f57d8bf-fvj8z" event={"ID":"9ccc665c-b410-4af4-ac83-e9ca0d632bcf","Type":"ContainerStarted","Data":"58170b97fe0713d2a59acdf5fc626054c3800a840b5489688dc848568034ef39"} Jul 29 04:04:50.999291 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:50.999242 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7b9b86fd-tst67" event={"ID":"8d0b6a0c-2568-41c7-bef8-a32dee9ca796","Type":"ContainerStarted","Data":"b7bd1c0f587fed9e5b877f4f3fe63d4814801e527f017cdd7198197930e72eac"} Jul 29 04:04:50.999450 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:50.999430 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7b9b86fd-tst67" Jul 29 04:04:51.000813 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:51.000790 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b9zpd" event={"ID":"3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b","Type":"ContainerStarted","Data":"adea1f941792bcb1dce03b7fef4b3f1d54fd1fe6831b9c91451327e6eb324fd6"} Jul 29 04:04:51.001409 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:51.001389 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7b9b86fd-tst67" Jul 29 04:04:51.003772 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:51.003747 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-56kwm" event={"ID":"1deee91e-bb4a-4511-953d-9fd13697bdc1","Type":"ContainerStarted","Data":"e98c98041abf723795e865265ed074a2459eefdc249ebf98349aef6ade5b910e"} Jul 29 04:04:51.008369 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:51.008318 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56447948cd-tvw7p" podStartSLOduration=2.476222921 podStartE2EDuration="11.008304346s" totalImagesPullingTime="8.532081425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:04:51.007631446 +0000 UTC m=+71.867006170" watchObservedRunningTime="2026-07-29 04:04:51.008304346 +0000 UTC m=+71.867679067" Jul 29 04:04:51.027795 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:51.027714 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b7b9b86fd-tst67" podStartSLOduration=2.245388009 podStartE2EDuration="11.027696184s" totalImagesPullingTime="8.782308175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:04:51.02467826 +0000 UTC m=+71.884052986" watchObservedRunningTime="2026-07-29 04:04:51.027696184 +0000 UTC m=+71.887070990" Jul 29 04:04:51.044227 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:51.044043 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-56kwm" podStartSLOduration=17.795228009 podStartE2EDuration="35.044023379s" totalImagesPullingTime="17.24879537s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 04:04:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:04:51.042611146 +0000 UTC m=+71.901985863" watchObservedRunningTime="2026-07-29 04:04:51.044023379 +0000 UTC m=+71.903398104" Jul 29 04:04:51.060781 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:51.059740 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-589f57d8bf-fvj8z" podStartSLOduration=2.5316843369999997 podStartE2EDuration="11.0597165s" totalImagesPullingTime="8.528032163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:04:51.058555171 +0000 UTC m=+71.917929917" watchObservedRunningTime="2026-07-29 04:04:51.0597165 +0000 UTC m=+71.919091225" Jul 29 04:04:52.019100 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:52.018686 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zbrk8" event={"ID":"9fa6e244-be92-4245-a54e-7aa303802f4b","Type":"ContainerStarted","Data":"cc8f4f7e835d90201618b938b05e6aa5f563f14bc3154a996553f803bc7be356"} Jul 29 04:04:52.019100 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:52.018769 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zbrk8" event={"ID":"9fa6e244-be92-4245-a54e-7aa303802f4b","Type":"ContainerStarted","Data":"4f295ccc4d3d247f3d4b1166a29fa3328b5c4ac8e5053e61cd432467a0074f86"} Jul 29 04:04:52.034964 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:52.034915 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zbrk8" podStartSLOduration=71.66427861 podStartE2EDuration="1m13.034897408s" totalImagesPullingTime="1.370618798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:03:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:04:52.034366836 +0000 UTC m=+72.893741561" watchObservedRunningTime="2026-07-29 04:04:52.034897408 +0000 UTC m=+72.894272132" Jul 29 04:04:55.032851 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:55.032811 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b9zpd" event={"ID":"3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b","Type":"ContainerStarted","Data":"66be80ae830b673e3372e590e240d5efa857d5f47119093334c4b6afd62cbe74"} Jul 29 04:04:55.034445 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:55.034419 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x6vhn" event={"ID":"350381e6-34a2-4f35-90d4-460ed24c7096","Type":"ContainerStarted","Data":"b2a9f5c86f17ca9acba4ac5003d82de6810eb4d9f7d5b17aa3e75d3d7e92370b"} Jul 29 04:04:55.034571 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:55.034555 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:04:55.048354 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:55.048310 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-b9zpd" podStartSLOduration=71.583334784 podStartE2EDuration="1m16.048297677s" totalImagesPullingTime="4.464962893s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 04:03:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:04:55.047751769 +0000 UTC m=+75.907126494" watchObservedRunningTime="2026-07-29 04:04:55.048297677 +0000 UTC m=+75.907672400" Jul 29 04:04:55.064900 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:04:55.064814 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-x6vhn" podStartSLOduration=71.607240164 podStartE2EDuration="1m16.06479666s" totalImagesPullingTime="4.457556496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:03:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:04:55.063434569 +0000 UTC m=+75.922809306" watchObservedRunningTime="2026-07-29 04:04:55.06479666 +0000 UTC m=+75.924171387" Jul 29 04:05:00.064024 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.063992 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-27rsx"] Jul 29 04:05:00.067029 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.067013 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-27rsx" Jul 29 04:05:00.069310 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.069279 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 29 04:05:00.069310 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.069283 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 29 04:05:00.069577 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.069561 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 29 04:05:00.069809 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.069786 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 29 04:05:00.069914 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.069790 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 29 04:05:00.070109 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.070095 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dqgcw\"" Jul 29 04:05:00.070412 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.070397 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 29 04:05:00.222172 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.222130 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/64605a73-fc63-46da-ba97-768a1215cd86-node-exporter-tls\") pod \"node-exporter-27rsx\" (UID: \"64605a73-fc63-46da-ba97-768a1215cd86\") " pod="openshift-monitoring/node-exporter-27rsx" Jul 29 04:05:00.222362 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.222186 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqg2r\" (UniqueName: \"kubernetes.io/projected/64605a73-fc63-46da-ba97-768a1215cd86-kube-api-access-wqg2r\") pod \"node-exporter-27rsx\" (UID: \"64605a73-fc63-46da-ba97-768a1215cd86\") " pod="openshift-monitoring/node-exporter-27rsx" Jul 29 04:05:00.222362 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.222234 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/64605a73-fc63-46da-ba97-768a1215cd86-sys\") pod \"node-exporter-27rsx\" (UID: \"64605a73-fc63-46da-ba97-768a1215cd86\") " pod="openshift-monitoring/node-exporter-27rsx" Jul 29 04:05:00.222362 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.222272 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/64605a73-fc63-46da-ba97-768a1215cd86-node-exporter-wtmp\") pod \"node-exporter-27rsx\" (UID: \"64605a73-fc63-46da-ba97-768a1215cd86\") " pod="openshift-monitoring/node-exporter-27rsx" Jul 29 04:05:00.222362 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.222299 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/64605a73-fc63-46da-ba97-768a1215cd86-node-exporter-accelerators-collector-config\") pod \"node-exporter-27rsx\" (UID: \"64605a73-fc63-46da-ba97-768a1215cd86\") " pod="openshift-monitoring/node-exporter-27rsx" Jul 29 04:05:00.222362 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.222345 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/64605a73-fc63-46da-ba97-768a1215cd86-root\") pod \"node-exporter-27rsx\" (UID: \"64605a73-fc63-46da-ba97-768a1215cd86\") " pod="openshift-monitoring/node-exporter-27rsx" Jul 29 04:05:00.222609 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.222367 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64605a73-fc63-46da-ba97-768a1215cd86-metrics-client-ca\") pod \"node-exporter-27rsx\" (UID: \"64605a73-fc63-46da-ba97-768a1215cd86\") " pod="openshift-monitoring/node-exporter-27rsx" Jul 29 04:05:00.222609 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.222394 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64605a73-fc63-46da-ba97-768a1215cd86-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-27rsx\" (UID: \"64605a73-fc63-46da-ba97-768a1215cd86\") " pod="openshift-monitoring/node-exporter-27rsx" Jul 29 04:05:00.222609 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.222486 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/64605a73-fc63-46da-ba97-768a1215cd86-node-exporter-textfile\") pod \"node-exporter-27rsx\" (UID: \"64605a73-fc63-46da-ba97-768a1215cd86\") " pod="openshift-monitoring/node-exporter-27rsx" Jul 29 04:05:00.224009 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.223983 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:05:00.323585 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.323495 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/64605a73-fc63-46da-ba97-768a1215cd86-node-exporter-textfile\") pod \"node-exporter-27rsx\" (UID: \"64605a73-fc63-46da-ba97-768a1215cd86\") " pod="openshift-monitoring/node-exporter-27rsx" Jul 29 04:05:00.323585 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.323552 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/64605a73-fc63-46da-ba97-768a1215cd86-node-exporter-tls\") pod \"node-exporter-27rsx\" (UID: \"64605a73-fc63-46da-ba97-768a1215cd86\") " pod="openshift-monitoring/node-exporter-27rsx" Jul 29 04:05:00.323585 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.323581 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqg2r\" (UniqueName: \"kubernetes.io/projected/64605a73-fc63-46da-ba97-768a1215cd86-kube-api-access-wqg2r\") pod \"node-exporter-27rsx\" (UID: \"64605a73-fc63-46da-ba97-768a1215cd86\") " pod="openshift-monitoring/node-exporter-27rsx" Jul 29 04:05:00.323848 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.323607 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/64605a73-fc63-46da-ba97-768a1215cd86-sys\") pod \"node-exporter-27rsx\" (UID: \"64605a73-fc63-46da-ba97-768a1215cd86\") " pod="openshift-monitoring/node-exporter-27rsx" Jul 29 04:05:00.323848 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.323641 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/64605a73-fc63-46da-ba97-768a1215cd86-node-exporter-wtmp\") pod \"node-exporter-27rsx\" (UID: \"64605a73-fc63-46da-ba97-768a1215cd86\") " pod="openshift-monitoring/node-exporter-27rsx" Jul 29 04:05:00.323848 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.323789 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/64605a73-fc63-46da-ba97-768a1215cd86-node-exporter-wtmp\") pod \"node-exporter-27rsx\" (UID: \"64605a73-fc63-46da-ba97-768a1215cd86\") " pod="openshift-monitoring/node-exporter-27rsx" Jul 29 04:05:00.324008 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.323859 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/64605a73-fc63-46da-ba97-768a1215cd86-sys\") pod \"node-exporter-27rsx\" (UID: \"64605a73-fc63-46da-ba97-768a1215cd86\") " pod="openshift-monitoring/node-exporter-27rsx" Jul 29 04:05:00.324008 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.323875 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/64605a73-fc63-46da-ba97-768a1215cd86-node-exporter-textfile\") pod \"node-exporter-27rsx\" (UID: \"64605a73-fc63-46da-ba97-768a1215cd86\") " pod="openshift-monitoring/node-exporter-27rsx" Jul 29 04:05:00.324008 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.323914 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/64605a73-fc63-46da-ba97-768a1215cd86-node-exporter-accelerators-collector-config\") pod \"node-exporter-27rsx\" (UID: \"64605a73-fc63-46da-ba97-768a1215cd86\") " pod="openshift-monitoring/node-exporter-27rsx" Jul 29 04:05:00.324008 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.323963 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/64605a73-fc63-46da-ba97-768a1215cd86-root\") pod \"node-exporter-27rsx\" (UID: \"64605a73-fc63-46da-ba97-768a1215cd86\") " pod="openshift-monitoring/node-exporter-27rsx" Jul 29 04:05:00.324008 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.323988 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64605a73-fc63-46da-ba97-768a1215cd86-metrics-client-ca\") pod \"node-exporter-27rsx\" (UID: \"64605a73-fc63-46da-ba97-768a1215cd86\") " pod="openshift-monitoring/node-exporter-27rsx" Jul 29 04:05:00.324189 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.324012 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/64605a73-fc63-46da-ba97-768a1215cd86-root\") pod \"node-exporter-27rsx\" (UID: \"64605a73-fc63-46da-ba97-768a1215cd86\") " pod="openshift-monitoring/node-exporter-27rsx" Jul 29 04:05:00.324189 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.324018 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64605a73-fc63-46da-ba97-768a1215cd86-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-27rsx\" (UID: \"64605a73-fc63-46da-ba97-768a1215cd86\") " pod="openshift-monitoring/node-exporter-27rsx" Jul 29 04:05:00.324587 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.324564 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/64605a73-fc63-46da-ba97-768a1215cd86-node-exporter-accelerators-collector-config\") pod \"node-exporter-27rsx\" (UID: \"64605a73-fc63-46da-ba97-768a1215cd86\") " pod="openshift-monitoring/node-exporter-27rsx" Jul 29 04:05:00.324678 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.324618 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64605a73-fc63-46da-ba97-768a1215cd86-metrics-client-ca\") pod \"node-exporter-27rsx\" (UID: \"64605a73-fc63-46da-ba97-768a1215cd86\") " pod="openshift-monitoring/node-exporter-27rsx" Jul 29 04:05:00.326119 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.326099 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/64605a73-fc63-46da-ba97-768a1215cd86-node-exporter-tls\") pod \"node-exporter-27rsx\" (UID: \"64605a73-fc63-46da-ba97-768a1215cd86\") " pod="openshift-monitoring/node-exporter-27rsx" Jul 29 04:05:00.326266 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.326178 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64605a73-fc63-46da-ba97-768a1215cd86-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-27rsx\" (UID: \"64605a73-fc63-46da-ba97-768a1215cd86\") " pod="openshift-monitoring/node-exporter-27rsx" Jul 29 04:05:00.331158 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.331115 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqg2r\" (UniqueName: \"kubernetes.io/projected/64605a73-fc63-46da-ba97-768a1215cd86-kube-api-access-wqg2r\") pod \"node-exporter-27rsx\" (UID: \"64605a73-fc63-46da-ba97-768a1215cd86\") " pod="openshift-monitoring/node-exporter-27rsx" Jul 29 04:05:00.376879 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:00.376849 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-27rsx" Jul 29 04:05:00.389982 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:05:00.389949 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64605a73_fc63_46da_ba97_768a1215cd86.slice/crio-bb9294ed0bfbf5867bbd943c9bb634045bdd043e48f2d849767242d492a1fb22 WatchSource:0}: Error finding container bb9294ed0bfbf5867bbd943c9bb634045bdd043e48f2d849767242d492a1fb22: Status 404 returned error can't find the container with id bb9294ed0bfbf5867bbd943c9bb634045bdd043e48f2d849767242d492a1fb22 Jul 29 04:05:01.057355 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:01.057316 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-27rsx" event={"ID":"64605a73-fc63-46da-ba97-768a1215cd86","Type":"ContainerStarted","Data":"bb9294ed0bfbf5867bbd943c9bb634045bdd043e48f2d849767242d492a1fb22"} Jul 29 04:05:02.063786 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:02.063755 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-27rsx" event={"ID":"64605a73-fc63-46da-ba97-768a1215cd86","Type":"ContainerStarted","Data":"61b0d2284c05adfd4ed3572eeeb4d7fd36ae2004f9e60d0f5c55ed35dd35f15b"} Jul 29 04:05:02.954234 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:02.954184 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-bbfb64c88-lv4w2" Jul 29 04:05:03.069122 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:03.069085 2579 generic.go:356] "Generic (PLEG): container finished" podID="64605a73-fc63-46da-ba97-768a1215cd86" containerID="61b0d2284c05adfd4ed3572eeeb4d7fd36ae2004f9e60d0f5c55ed35dd35f15b" exitCode=0 Jul 29 04:05:03.069528 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:03.069171 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-27rsx" event={"ID":"64605a73-fc63-46da-ba97-768a1215cd86","Type":"ContainerDied","Data":"61b0d2284c05adfd4ed3572eeeb4d7fd36ae2004f9e60d0f5c55ed35dd35f15b"} Jul 29 04:05:04.075575 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:04.075542 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-27rsx" event={"ID":"64605a73-fc63-46da-ba97-768a1215cd86","Type":"ContainerStarted","Data":"f00fa48e9ffd5a20f943ee8a9a19c9fd2583096c0dc6b47d5f6279e5bdad2874"} Jul 29 04:05:04.075934 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:04.075596 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-27rsx" event={"ID":"64605a73-fc63-46da-ba97-768a1215cd86","Type":"ContainerStarted","Data":"5da9452989c4c81b621449e11b2f95c5c194c4931b87b183de26a23ce94b63e0"} Jul 29 04:05:04.094725 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:04.094668 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-27rsx" podStartSLOduration=2.497163647 podStartE2EDuration="4.094649264s" totalImagesPullingTime="1.533125366s" totalInitContainerRuntime="64.360251ms" isStatefulPod=true podCreationTimestamp="2026-07-29 04:05:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:05:04.092733855 +0000 UTC m=+84.952108580" watchObservedRunningTime="2026-07-29 04:05:04.094649264 +0000 UTC m=+84.954023989" Jul 29 04:05:05.239010 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:05.238961 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" podUID="f194f797-9d02-465f-9293-e5e6d4226a21" containerName="registry" containerID="cri-o://bd68d37a6efb92deaf529364c904104bf5d023cfb7f048b77fbd7775aa3d3544" gracePeriod=30 Jul 29 04:05:05.481217 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:05.481178 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:05:05.569488 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:05.569391 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f194f797-9d02-465f-9293-e5e6d4226a21-registry-certificates\") pod \"f194f797-9d02-465f-9293-e5e6d4226a21\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " Jul 29 04:05:05.569488 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:05.569441 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f194f797-9d02-465f-9293-e5e6d4226a21-ca-trust-extracted\") pod \"f194f797-9d02-465f-9293-e5e6d4226a21\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " Jul 29 04:05:05.569488 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:05.569469 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f194f797-9d02-465f-9293-e5e6d4226a21-installation-pull-secrets\") pod \"f194f797-9d02-465f-9293-e5e6d4226a21\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " Jul 29 04:05:05.569488 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:05.569486 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-registry-tls\") pod \"f194f797-9d02-465f-9293-e5e6d4226a21\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " Jul 29 04:05:05.569811 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:05.569506 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f194f797-9d02-465f-9293-e5e6d4226a21-trusted-ca\") pod \"f194f797-9d02-465f-9293-e5e6d4226a21\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " Jul 29 04:05:05.569811 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:05.569544 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-bound-sa-token\") pod \"f194f797-9d02-465f-9293-e5e6d4226a21\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " Jul 29 04:05:05.569811 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:05.569585 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f194f797-9d02-465f-9293-e5e6d4226a21-image-registry-private-configuration\") pod \"f194f797-9d02-465f-9293-e5e6d4226a21\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " Jul 29 04:05:05.569811 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:05.569612 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4pbf\" (UniqueName: \"kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-kube-api-access-b4pbf\") pod \"f194f797-9d02-465f-9293-e5e6d4226a21\" (UID: \"f194f797-9d02-465f-9293-e5e6d4226a21\") " Jul 29 04:05:05.570003 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:05.569960 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f194f797-9d02-465f-9293-e5e6d4226a21-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f194f797-9d02-465f-9293-e5e6d4226a21" (UID: "f194f797-9d02-465f-9293-e5e6d4226a21"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 04:05:05.570505 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:05.570479 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f194f797-9d02-465f-9293-e5e6d4226a21-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f194f797-9d02-465f-9293-e5e6d4226a21" (UID: "f194f797-9d02-465f-9293-e5e6d4226a21"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 04:05:05.572455 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:05.572427 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f194f797-9d02-465f-9293-e5e6d4226a21" (UID: "f194f797-9d02-465f-9293-e5e6d4226a21"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:05:05.572583 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:05.572486 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f194f797-9d02-465f-9293-e5e6d4226a21-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f194f797-9d02-465f-9293-e5e6d4226a21" (UID: "f194f797-9d02-465f-9293-e5e6d4226a21"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 04:05:05.572633 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:05.572588 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-kube-api-access-b4pbf" (OuterVolumeSpecName: "kube-api-access-b4pbf") pod "f194f797-9d02-465f-9293-e5e6d4226a21" (UID: "f194f797-9d02-465f-9293-e5e6d4226a21"). InnerVolumeSpecName "kube-api-access-b4pbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:05:05.572633 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:05.572616 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f194f797-9d02-465f-9293-e5e6d4226a21-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f194f797-9d02-465f-9293-e5e6d4226a21" (UID: "f194f797-9d02-465f-9293-e5e6d4226a21"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 04:05:05.572692 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:05.572654 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f194f797-9d02-465f-9293-e5e6d4226a21" (UID: "f194f797-9d02-465f-9293-e5e6d4226a21"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:05:05.578510 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:05.578486 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f194f797-9d02-465f-9293-e5e6d4226a21-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f194f797-9d02-465f-9293-e5e6d4226a21" (UID: "f194f797-9d02-465f-9293-e5e6d4226a21"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 04:05:05.670772 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:05.670733 2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f194f797-9d02-465f-9293-e5e6d4226a21-installation-pull-secrets\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:05:05.670772 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:05.670765 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-registry-tls\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:05:05.670772 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:05.670776 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f194f797-9d02-465f-9293-e5e6d4226a21-trusted-ca\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:05:05.670772 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:05.670783 2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-bound-sa-token\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:05:05.671029 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:05.670792 2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f194f797-9d02-465f-9293-e5e6d4226a21-image-registry-private-configuration\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:05:05.671029 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:05.670801 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4pbf\" (UniqueName: \"kubernetes.io/projected/f194f797-9d02-465f-9293-e5e6d4226a21-kube-api-access-b4pbf\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:05:05.671029 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:05.670809 2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f194f797-9d02-465f-9293-e5e6d4226a21-registry-certificates\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:05:05.671029 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:05.670817 2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f194f797-9d02-465f-9293-e5e6d4226a21-ca-trust-extracted\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:05:06.082662 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:06.082629 2579 generic.go:356] "Generic (PLEG): container finished" podID="f194f797-9d02-465f-9293-e5e6d4226a21" containerID="bd68d37a6efb92deaf529364c904104bf5d023cfb7f048b77fbd7775aa3d3544" exitCode=0 Jul 29 04:05:06.082841 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:06.082707 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" Jul 29 04:05:06.082841 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:06.082723 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" event={"ID":"f194f797-9d02-465f-9293-e5e6d4226a21","Type":"ContainerDied","Data":"bd68d37a6efb92deaf529364c904104bf5d023cfb7f048b77fbd7775aa3d3544"} Jul 29 04:05:06.082841 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:06.082776 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55d9f6d798-l5rf7" event={"ID":"f194f797-9d02-465f-9293-e5e6d4226a21","Type":"ContainerDied","Data":"f5ca3c0c251514061be10f0533823367709b3d2b1b5b7b92c0a4ab95557f40d2"} Jul 29 04:05:06.082841 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:06.082798 2579 scope.go:117] "RemoveContainer" containerID="bd68d37a6efb92deaf529364c904104bf5d023cfb7f048b77fbd7775aa3d3544" Jul 29 04:05:06.090966 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:06.090947 2579 scope.go:117] "RemoveContainer" containerID="bd68d37a6efb92deaf529364c904104bf5d023cfb7f048b77fbd7775aa3d3544" Jul 29 04:05:06.091298 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:05:06.091278 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd68d37a6efb92deaf529364c904104bf5d023cfb7f048b77fbd7775aa3d3544\": container with ID starting with bd68d37a6efb92deaf529364c904104bf5d023cfb7f048b77fbd7775aa3d3544 not found: ID does not exist" containerID="bd68d37a6efb92deaf529364c904104bf5d023cfb7f048b77fbd7775aa3d3544" Jul 29 04:05:06.091354 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:06.091307 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd68d37a6efb92deaf529364c904104bf5d023cfb7f048b77fbd7775aa3d3544"} err="failed to get container status \"bd68d37a6efb92deaf529364c904104bf5d023cfb7f048b77fbd7775aa3d3544\": rpc error: code = NotFound desc = could not find container \"bd68d37a6efb92deaf529364c904104bf5d023cfb7f048b77fbd7775aa3d3544\": container with ID starting with bd68d37a6efb92deaf529364c904104bf5d023cfb7f048b77fbd7775aa3d3544 not found: ID does not exist" Jul 29 04:05:06.098582 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:06.098562 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55d9f6d798-l5rf7"] Jul 29 04:05:06.101272 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:06.101252 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-55d9f6d798-l5rf7"] Jul 29 04:05:07.590036 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:07.590006 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f194f797-9d02-465f-9293-e5e6d4226a21" path="/var/lib/kubelet/pods/f194f797-9d02-465f-9293-e5e6d4226a21/volumes" Jul 29 04:05:26.040785 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:26.040754 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-x6vhn" Jul 29 04:05:31.653029 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:05:31.653005 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bprlw_dcdad901-fa84-42eb-8e6d-70c86d2639cb/serve-healthcheck-canary/0.log" Jul 29 04:08:39.460132 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:39.460105 2579 kubelet.go:1648] "Image garbage collection succeeded" Jul 29 04:08:39.925418 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:39.925341 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64c765c8-p2bl9"] Jul 29 04:08:39.927342 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:39.925592 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f194f797-9d02-465f-9293-e5e6d4226a21" containerName="registry" Jul 29 04:08:39.927342 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:39.925603 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f194f797-9d02-465f-9293-e5e6d4226a21" containerName="registry" Jul 29 04:08:39.927342 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:39.925652 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f194f797-9d02-465f-9293-e5e6d4226a21" containerName="registry" Jul 29 04:08:39.928429 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:39.928413 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64c765c8-p2bl9" Jul 29 04:08:39.931102 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:39.931043 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 29 04:08:39.931102 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:39.931063 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 29 04:08:39.931334 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:39.931065 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-6c7bk\"" Jul 29 04:08:39.931334 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:39.931308 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 29 04:08:39.931440 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:39.931389 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 29 04:08:39.945758 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:39.945730 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64c765c8-p2bl9"] Jul 29 04:08:40.071062 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:40.071021 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4a99836-ced3-435f-8498-5c0ae65134cc-tmp\") pod \"opendatahub-operator-controller-manager-64c765c8-p2bl9\" (UID: \"e4a99836-ced3-435f-8498-5c0ae65134cc\") " pod="opendatahub/opendatahub-operator-controller-manager-64c765c8-p2bl9" Jul 29 04:08:40.071276 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:40.071082 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e4a99836-ced3-435f-8498-5c0ae65134cc-webhook-cert\") pod \"opendatahub-operator-controller-manager-64c765c8-p2bl9\" (UID: \"e4a99836-ced3-435f-8498-5c0ae65134cc\") " pod="opendatahub/opendatahub-operator-controller-manager-64c765c8-p2bl9" Jul 29 04:08:40.071276 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:40.071139 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdhkl\" (UniqueName: \"kubernetes.io/projected/e4a99836-ced3-435f-8498-5c0ae65134cc-kube-api-access-mdhkl\") pod \"opendatahub-operator-controller-manager-64c765c8-p2bl9\" (UID: \"e4a99836-ced3-435f-8498-5c0ae65134cc\") " pod="opendatahub/opendatahub-operator-controller-manager-64c765c8-p2bl9" Jul 29 04:08:40.071276 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:40.071190 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e4a99836-ced3-435f-8498-5c0ae65134cc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64c765c8-p2bl9\" (UID: \"e4a99836-ced3-435f-8498-5c0ae65134cc\") " pod="opendatahub/opendatahub-operator-controller-manager-64c765c8-p2bl9" Jul 29 04:08:40.071435 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:40.071289 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e4a99836-ced3-435f-8498-5c0ae65134cc-manifests\") pod \"opendatahub-operator-controller-manager-64c765c8-p2bl9\" (UID: \"e4a99836-ced3-435f-8498-5c0ae65134cc\") " pod="opendatahub/opendatahub-operator-controller-manager-64c765c8-p2bl9" Jul 29 04:08:40.172195 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:40.172159 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4a99836-ced3-435f-8498-5c0ae65134cc-tmp\") pod \"opendatahub-operator-controller-manager-64c765c8-p2bl9\" (UID: \"e4a99836-ced3-435f-8498-5c0ae65134cc\") " pod="opendatahub/opendatahub-operator-controller-manager-64c765c8-p2bl9" Jul 29 04:08:40.172195 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:40.172220 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e4a99836-ced3-435f-8498-5c0ae65134cc-webhook-cert\") pod \"opendatahub-operator-controller-manager-64c765c8-p2bl9\" (UID: \"e4a99836-ced3-435f-8498-5c0ae65134cc\") " pod="opendatahub/opendatahub-operator-controller-manager-64c765c8-p2bl9" Jul 29 04:08:40.172451 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:40.172244 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdhkl\" (UniqueName: \"kubernetes.io/projected/e4a99836-ced3-435f-8498-5c0ae65134cc-kube-api-access-mdhkl\") pod \"opendatahub-operator-controller-manager-64c765c8-p2bl9\" (UID: \"e4a99836-ced3-435f-8498-5c0ae65134cc\") " pod="opendatahub/opendatahub-operator-controller-manager-64c765c8-p2bl9" Jul 29 04:08:40.172451 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:40.172279 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e4a99836-ced3-435f-8498-5c0ae65134cc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64c765c8-p2bl9\" (UID: \"e4a99836-ced3-435f-8498-5c0ae65134cc\") " pod="opendatahub/opendatahub-operator-controller-manager-64c765c8-p2bl9" Jul 29 04:08:40.172451 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:40.172329 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e4a99836-ced3-435f-8498-5c0ae65134cc-manifests\") pod \"opendatahub-operator-controller-manager-64c765c8-p2bl9\" (UID: \"e4a99836-ced3-435f-8498-5c0ae65134cc\") " pod="opendatahub/opendatahub-operator-controller-manager-64c765c8-p2bl9" Jul 29 04:08:40.172693 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:40.172670 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e4a99836-ced3-435f-8498-5c0ae65134cc-manifests\") pod \"opendatahub-operator-controller-manager-64c765c8-p2bl9\" (UID: \"e4a99836-ced3-435f-8498-5c0ae65134cc\") " pod="opendatahub/opendatahub-operator-controller-manager-64c765c8-p2bl9" Jul 29 04:08:40.174485 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:40.174457 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4a99836-ced3-435f-8498-5c0ae65134cc-tmp\") pod \"opendatahub-operator-controller-manager-64c765c8-p2bl9\" (UID: \"e4a99836-ced3-435f-8498-5c0ae65134cc\") " pod="opendatahub/opendatahub-operator-controller-manager-64c765c8-p2bl9" Jul 29 04:08:40.174632 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:40.174613 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e4a99836-ced3-435f-8498-5c0ae65134cc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64c765c8-p2bl9\" (UID: \"e4a99836-ced3-435f-8498-5c0ae65134cc\") " pod="opendatahub/opendatahub-operator-controller-manager-64c765c8-p2bl9" Jul 29 04:08:40.174799 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:40.174763 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e4a99836-ced3-435f-8498-5c0ae65134cc-webhook-cert\") pod \"opendatahub-operator-controller-manager-64c765c8-p2bl9\" (UID: \"e4a99836-ced3-435f-8498-5c0ae65134cc\") " pod="opendatahub/opendatahub-operator-controller-manager-64c765c8-p2bl9" Jul 29 04:08:40.181081 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:40.181057 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdhkl\" (UniqueName: \"kubernetes.io/projected/e4a99836-ced3-435f-8498-5c0ae65134cc-kube-api-access-mdhkl\") pod \"opendatahub-operator-controller-manager-64c765c8-p2bl9\" (UID: \"e4a99836-ced3-435f-8498-5c0ae65134cc\") " pod="opendatahub/opendatahub-operator-controller-manager-64c765c8-p2bl9" Jul 29 04:08:40.239840 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:40.239811 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64c765c8-p2bl9" Jul 29 04:08:40.379138 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:40.379106 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64c765c8-p2bl9"] Jul 29 04:08:40.384058 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:08:40.384029 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4a99836_ced3_435f_8498_5c0ae65134cc.slice/crio-002c29ea4ecda63be439dfe7e08e95cb98597740ea225bc59e20f192c35b2d19 WatchSource:0}: Error finding container 002c29ea4ecda63be439dfe7e08e95cb98597740ea225bc59e20f192c35b2d19: Status 404 returned error can't find the container with id 002c29ea4ecda63be439dfe7e08e95cb98597740ea225bc59e20f192c35b2d19 Jul 29 04:08:40.789904 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:40.789864 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64c765c8-p2bl9" event={"ID":"e4a99836-ced3-435f-8498-5c0ae65134cc","Type":"ContainerStarted","Data":"002c29ea4ecda63be439dfe7e08e95cb98597740ea225bc59e20f192c35b2d19"} Jul 29 04:08:43.802838 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:43.802801 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64c765c8-p2bl9" event={"ID":"e4a99836-ced3-435f-8498-5c0ae65134cc","Type":"ContainerStarted","Data":"9e900b0e59720765a881bcd148a9b8b99a981433c96b4312818edc00d037e02b"} Jul 29 04:08:44.807745 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:44.807662 2579 generic.go:356] "Generic (PLEG): container finished" podID="e4a99836-ced3-435f-8498-5c0ae65134cc" containerID="9e900b0e59720765a881bcd148a9b8b99a981433c96b4312818edc00d037e02b" exitCode=0 Jul 29 04:08:44.807745 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:44.807699 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64c765c8-p2bl9" event={"ID":"e4a99836-ced3-435f-8498-5c0ae65134cc","Type":"ContainerDied","Data":"9e900b0e59720765a881bcd148a9b8b99a981433c96b4312818edc00d037e02b"} Jul 29 04:08:44.808198 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:44.808181 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 04:08:45.812886 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:45.812852 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64c765c8-p2bl9" event={"ID":"e4a99836-ced3-435f-8498-5c0ae65134cc","Type":"ContainerStarted","Data":"caebe47d66374398f8b302db3f7056b849b6a7a743a407f8f931a71a03a93073"} Jul 29 04:08:45.813299 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:45.812955 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-64c765c8-p2bl9" Jul 29 04:08:45.834543 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:45.834494 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-64c765c8-p2bl9" podStartSLOduration=2.417431658 podStartE2EDuration="6.834477877s" totalImagesPullingTime="3.339464181s" totalInitContainerRuntime="1.077582038s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:08:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:08:45.832557762 +0000 UTC m=+306.691932486" watchObservedRunningTime="2026-07-29 04:08:45.834477877 +0000 UTC m=+306.693852654" Jul 29 04:08:56.819639 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:56.819602 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-64c765c8-p2bl9" Jul 29 04:08:59.527475 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:59.527439 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-76ccb86b9-lh4m6"] Jul 29 04:08:59.541727 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:59.541700 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-76ccb86b9-lh4m6" Jul 29 04:08:59.544539 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:59.544504 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 29 04:08:59.544686 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:59.544570 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-68q26\"" Jul 29 04:08:59.545681 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:59.545662 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 29 04:08:59.545788 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:59.545683 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 29 04:08:59.545850 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:59.545690 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 29 04:08:59.548408 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:59.548385 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-76ccb86b9-lh4m6"] Jul 29 04:08:59.601019 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:59.600982 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df92f309-917b-4659-a2a8-6e9a1baa70f9-tls-certs\") pod \"kube-auth-proxy-76ccb86b9-lh4m6\" (UID: \"df92f309-917b-4659-a2a8-6e9a1baa70f9\") " pod="openshift-ingress/kube-auth-proxy-76ccb86b9-lh4m6" Jul 29 04:08:59.601193 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:59.601059 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df92f309-917b-4659-a2a8-6e9a1baa70f9-tmp\") pod \"kube-auth-proxy-76ccb86b9-lh4m6\" (UID: \"df92f309-917b-4659-a2a8-6e9a1baa70f9\") " pod="openshift-ingress/kube-auth-proxy-76ccb86b9-lh4m6" Jul 29 04:08:59.601193 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:59.601088 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwjgq\" (UniqueName: \"kubernetes.io/projected/df92f309-917b-4659-a2a8-6e9a1baa70f9-kube-api-access-kwjgq\") pod \"kube-auth-proxy-76ccb86b9-lh4m6\" (UID: \"df92f309-917b-4659-a2a8-6e9a1baa70f9\") " pod="openshift-ingress/kube-auth-proxy-76ccb86b9-lh4m6" Jul 29 04:08:59.702142 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:59.702098 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df92f309-917b-4659-a2a8-6e9a1baa70f9-tls-certs\") pod \"kube-auth-proxy-76ccb86b9-lh4m6\" (UID: \"df92f309-917b-4659-a2a8-6e9a1baa70f9\") " pod="openshift-ingress/kube-auth-proxy-76ccb86b9-lh4m6" Jul 29 04:08:59.702350 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:59.702177 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df92f309-917b-4659-a2a8-6e9a1baa70f9-tmp\") pod \"kube-auth-proxy-76ccb86b9-lh4m6\" (UID: \"df92f309-917b-4659-a2a8-6e9a1baa70f9\") " pod="openshift-ingress/kube-auth-proxy-76ccb86b9-lh4m6" Jul 29 04:08:59.702350 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:59.702197 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwjgq\" (UniqueName: \"kubernetes.io/projected/df92f309-917b-4659-a2a8-6e9a1baa70f9-kube-api-access-kwjgq\") pod \"kube-auth-proxy-76ccb86b9-lh4m6\" (UID: \"df92f309-917b-4659-a2a8-6e9a1baa70f9\") " pod="openshift-ingress/kube-auth-proxy-76ccb86b9-lh4m6" Jul 29 04:08:59.704397 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:59.704368 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df92f309-917b-4659-a2a8-6e9a1baa70f9-tmp\") pod \"kube-auth-proxy-76ccb86b9-lh4m6\" (UID: \"df92f309-917b-4659-a2a8-6e9a1baa70f9\") " pod="openshift-ingress/kube-auth-proxy-76ccb86b9-lh4m6" Jul 29 04:08:59.704592 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:59.704575 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df92f309-917b-4659-a2a8-6e9a1baa70f9-tls-certs\") pod \"kube-auth-proxy-76ccb86b9-lh4m6\" (UID: \"df92f309-917b-4659-a2a8-6e9a1baa70f9\") " pod="openshift-ingress/kube-auth-proxy-76ccb86b9-lh4m6" Jul 29 04:08:59.716162 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:59.716137 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwjgq\" (UniqueName: \"kubernetes.io/projected/df92f309-917b-4659-a2a8-6e9a1baa70f9-kube-api-access-kwjgq\") pod \"kube-auth-proxy-76ccb86b9-lh4m6\" (UID: \"df92f309-917b-4659-a2a8-6e9a1baa70f9\") " pod="openshift-ingress/kube-auth-proxy-76ccb86b9-lh4m6" Jul 29 04:08:59.776803 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:59.776772 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-76msd"] Jul 29 04:08:59.780189 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:59.780147 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-76msd" Jul 29 04:08:59.786178 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:59.786153 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-k242z\"" Jul 29 04:08:59.786319 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:59.786253 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 29 04:08:59.801575 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:59.801549 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-76msd"] Jul 29 04:08:59.803463 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:59.803439 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k27q9\" (UniqueName: \"kubernetes.io/projected/0ea18530-6257-4356-b3c7-bd18ed2f1e6c-kube-api-access-k27q9\") pod \"odh-model-controller-7998c67dc4-76msd\" (UID: \"0ea18530-6257-4356-b3c7-bd18ed2f1e6c\") " pod="opendatahub/odh-model-controller-7998c67dc4-76msd" Jul 29 04:08:59.803576 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:59.803482 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ea18530-6257-4356-b3c7-bd18ed2f1e6c-cert\") pod \"odh-model-controller-7998c67dc4-76msd\" (UID: \"0ea18530-6257-4356-b3c7-bd18ed2f1e6c\") " pod="opendatahub/odh-model-controller-7998c67dc4-76msd" Jul 29 04:08:59.852579 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:59.852546 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-76ccb86b9-lh4m6" Jul 29 04:08:59.904122 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:59.904096 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k27q9\" (UniqueName: \"kubernetes.io/projected/0ea18530-6257-4356-b3c7-bd18ed2f1e6c-kube-api-access-k27q9\") pod \"odh-model-controller-7998c67dc4-76msd\" (UID: \"0ea18530-6257-4356-b3c7-bd18ed2f1e6c\") " pod="opendatahub/odh-model-controller-7998c67dc4-76msd" Jul 29 04:08:59.904254 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:59.904163 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ea18530-6257-4356-b3c7-bd18ed2f1e6c-cert\") pod \"odh-model-controller-7998c67dc4-76msd\" (UID: \"0ea18530-6257-4356-b3c7-bd18ed2f1e6c\") " pod="opendatahub/odh-model-controller-7998c67dc4-76msd" Jul 29 04:08:59.904553 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:08:59.904530 2579 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 29 04:08:59.904656 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:08:59.904616 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ea18530-6257-4356-b3c7-bd18ed2f1e6c-cert podName:0ea18530-6257-4356-b3c7-bd18ed2f1e6c nodeName:}" failed. No retries permitted until 2026-07-29 04:09:00.404592283 +0000 UTC m=+321.263966997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0ea18530-6257-4356-b3c7-bd18ed2f1e6c-cert") pod "odh-model-controller-7998c67dc4-76msd" (UID: "0ea18530-6257-4356-b3c7-bd18ed2f1e6c") : secret "odh-model-controller-webhook-cert" not found Jul 29 04:08:59.916156 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:59.916123 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k27q9\" (UniqueName: \"kubernetes.io/projected/0ea18530-6257-4356-b3c7-bd18ed2f1e6c-kube-api-access-k27q9\") pod \"odh-model-controller-7998c67dc4-76msd\" (UID: \"0ea18530-6257-4356-b3c7-bd18ed2f1e6c\") " pod="opendatahub/odh-model-controller-7998c67dc4-76msd" Jul 29 04:08:59.996732 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:08:59.996669 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-76ccb86b9-lh4m6"] Jul 29 04:08:59.999725 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:08:59.999700 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf92f309_917b_4659_a2a8_6e9a1baa70f9.slice/crio-94fbe5695c41c12e98f0e46dbdecfd14c1b2a3b1d920dc80c619b9f3f846ac3f WatchSource:0}: Error finding container 94fbe5695c41c12e98f0e46dbdecfd14c1b2a3b1d920dc80c619b9f3f846ac3f: Status 404 returned error can't find the container with id 94fbe5695c41c12e98f0e46dbdecfd14c1b2a3b1d920dc80c619b9f3f846ac3f Jul 29 04:09:00.408658 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:00.408624 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ea18530-6257-4356-b3c7-bd18ed2f1e6c-cert\") pod \"odh-model-controller-7998c67dc4-76msd\" (UID: \"0ea18530-6257-4356-b3c7-bd18ed2f1e6c\") " pod="opendatahub/odh-model-controller-7998c67dc4-76msd" Jul 29 04:09:00.411001 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:00.410982 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ea18530-6257-4356-b3c7-bd18ed2f1e6c-cert\") pod \"odh-model-controller-7998c67dc4-76msd\" (UID: \"0ea18530-6257-4356-b3c7-bd18ed2f1e6c\") " pod="opendatahub/odh-model-controller-7998c67dc4-76msd" Jul 29 04:09:00.690373 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:00.690344 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-76msd" Jul 29 04:09:00.737237 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:00.735766 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ndfqk"] Jul 29 04:09:00.745577 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:00.745507 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ndfqk" Jul 29 04:09:00.748765 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:00.748675 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 29 04:09:00.777754 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:00.777692 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ndfqk"] Jul 29 04:09:00.812657 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:00.812624 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d1d3c77-b6f7-41c2-804c-cc829622aab0-tls-certs\") pod \"model-serving-api-54d89bd5fc-ndfqk\" (UID: \"0d1d3c77-b6f7-41c2-804c-cc829622aab0\") " pod="opendatahub/model-serving-api-54d89bd5fc-ndfqk" Jul 29 04:09:00.812838 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:00.812670 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4p5d\" (UniqueName: \"kubernetes.io/projected/0d1d3c77-b6f7-41c2-804c-cc829622aab0-kube-api-access-v4p5d\") pod \"model-serving-api-54d89bd5fc-ndfqk\" (UID: \"0d1d3c77-b6f7-41c2-804c-cc829622aab0\") " pod="opendatahub/model-serving-api-54d89bd5fc-ndfqk" Jul 29 04:09:00.865453 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:00.865401 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-76ccb86b9-lh4m6" event={"ID":"df92f309-917b-4659-a2a8-6e9a1baa70f9","Type":"ContainerStarted","Data":"94fbe5695c41c12e98f0e46dbdecfd14c1b2a3b1d920dc80c619b9f3f846ac3f"} Jul 29 04:09:00.866893 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:00.866730 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-76msd"] Jul 29 04:09:00.869676 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:09:00.869650 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ea18530_6257_4356_b3c7_bd18ed2f1e6c.slice/crio-6fd2f17c7472208ea2907981a8a3986bf8eacf67365edcdc2c6f60caa16c6989 WatchSource:0}: Error finding container 6fd2f17c7472208ea2907981a8a3986bf8eacf67365edcdc2c6f60caa16c6989: Status 404 returned error can't find the container with id 6fd2f17c7472208ea2907981a8a3986bf8eacf67365edcdc2c6f60caa16c6989 Jul 29 04:09:00.913915 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:00.913887 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d1d3c77-b6f7-41c2-804c-cc829622aab0-tls-certs\") pod \"model-serving-api-54d89bd5fc-ndfqk\" (UID: \"0d1d3c77-b6f7-41c2-804c-cc829622aab0\") " pod="opendatahub/model-serving-api-54d89bd5fc-ndfqk" Jul 29 04:09:00.914094 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:00.913922 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4p5d\" (UniqueName: \"kubernetes.io/projected/0d1d3c77-b6f7-41c2-804c-cc829622aab0-kube-api-access-v4p5d\") pod \"model-serving-api-54d89bd5fc-ndfqk\" (UID: \"0d1d3c77-b6f7-41c2-804c-cc829622aab0\") " pod="opendatahub/model-serving-api-54d89bd5fc-ndfqk" Jul 29 04:09:00.916817 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:00.916790 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d1d3c77-b6f7-41c2-804c-cc829622aab0-tls-certs\") pod \"model-serving-api-54d89bd5fc-ndfqk\" (UID: \"0d1d3c77-b6f7-41c2-804c-cc829622aab0\") " pod="opendatahub/model-serving-api-54d89bd5fc-ndfqk" Jul 29 04:09:00.926055 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:00.926017 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4p5d\" (UniqueName: \"kubernetes.io/projected/0d1d3c77-b6f7-41c2-804c-cc829622aab0-kube-api-access-v4p5d\") pod \"model-serving-api-54d89bd5fc-ndfqk\" (UID: \"0d1d3c77-b6f7-41c2-804c-cc829622aab0\") " pod="opendatahub/model-serving-api-54d89bd5fc-ndfqk" Jul 29 04:09:01.058651 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.058555 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ndfqk" Jul 29 04:09:01.143125 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.143093 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95"] Jul 29 04:09:01.147697 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.147368 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" Jul 29 04:09:01.150191 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.149973 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 29 04:09:01.150191 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.150016 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 29 04:09:01.150387 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.150284 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 29 04:09:01.150589 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.150568 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2qvd6\"" Jul 29 04:09:01.152388 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.152364 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 29 04:09:01.159014 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.158944 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95"] Jul 29 04:09:01.201357 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.201335 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ndfqk"] Jul 29 04:09:01.203687 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:09:01.203651 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d1d3c77_b6f7_41c2_804c_cc829622aab0.slice/crio-377ee9a0822aec0c71d6d582b73d19fa8cce08ef998c4f589c562effbc0cdf1a WatchSource:0}: Error finding container 377ee9a0822aec0c71d6d582b73d19fa8cce08ef998c4f589c562effbc0cdf1a: Status 404 returned error can't find the container with id 377ee9a0822aec0c71d6d582b73d19fa8cce08ef998c4f589c562effbc0cdf1a Jul 29 04:09:01.216490 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.216466 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdf36597-79ac-47ed-ba77-edcd447fdc53-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-qfh95\" (UID: \"bdf36597-79ac-47ed-ba77-edcd447fdc53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" Jul 29 04:09:01.216606 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.216501 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdf36597-79ac-47ed-ba77-edcd447fdc53-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-qfh95\" (UID: \"bdf36597-79ac-47ed-ba77-edcd447fdc53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" Jul 29 04:09:01.216606 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.216523 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcfh5\" (UniqueName: \"kubernetes.io/projected/bdf36597-79ac-47ed-ba77-edcd447fdc53-kube-api-access-mcfh5\") pod \"istiod-openshift-gateway-7b7995d5cd-qfh95\" (UID: \"bdf36597-79ac-47ed-ba77-edcd447fdc53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" Jul 29 04:09:01.216606 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.216582 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bdf36597-79ac-47ed-ba77-edcd447fdc53-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-qfh95\" (UID: \"bdf36597-79ac-47ed-ba77-edcd447fdc53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" Jul 29 04:09:01.216768 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.216614 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bdf36597-79ac-47ed-ba77-edcd447fdc53-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-qfh95\" (UID: \"bdf36597-79ac-47ed-ba77-edcd447fdc53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" Jul 29 04:09:01.216768 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.216642 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bdf36597-79ac-47ed-ba77-edcd447fdc53-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-qfh95\" (UID: \"bdf36597-79ac-47ed-ba77-edcd447fdc53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" Jul 29 04:09:01.216768 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.216701 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bdf36597-79ac-47ed-ba77-edcd447fdc53-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-qfh95\" (UID: \"bdf36597-79ac-47ed-ba77-edcd447fdc53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" Jul 29 04:09:01.317324 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.317247 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdf36597-79ac-47ed-ba77-edcd447fdc53-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-qfh95\" (UID: \"bdf36597-79ac-47ed-ba77-edcd447fdc53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" Jul 29 04:09:01.317324 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.317282 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcfh5\" (UniqueName: \"kubernetes.io/projected/bdf36597-79ac-47ed-ba77-edcd447fdc53-kube-api-access-mcfh5\") pod \"istiod-openshift-gateway-7b7995d5cd-qfh95\" (UID: \"bdf36597-79ac-47ed-ba77-edcd447fdc53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" Jul 29 04:09:01.317324 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.317302 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bdf36597-79ac-47ed-ba77-edcd447fdc53-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-qfh95\" (UID: \"bdf36597-79ac-47ed-ba77-edcd447fdc53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" Jul 29 04:09:01.317600 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.317330 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bdf36597-79ac-47ed-ba77-edcd447fdc53-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-qfh95\" (UID: \"bdf36597-79ac-47ed-ba77-edcd447fdc53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" Jul 29 04:09:01.317600 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.317446 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bdf36597-79ac-47ed-ba77-edcd447fdc53-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-qfh95\" (UID: \"bdf36597-79ac-47ed-ba77-edcd447fdc53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" Jul 29 04:09:01.317600 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.317524 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bdf36597-79ac-47ed-ba77-edcd447fdc53-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-qfh95\" (UID: \"bdf36597-79ac-47ed-ba77-edcd447fdc53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" Jul 29 04:09:01.317600 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.317549 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdf36597-79ac-47ed-ba77-edcd447fdc53-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-qfh95\" (UID: \"bdf36597-79ac-47ed-ba77-edcd447fdc53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" Jul 29 04:09:01.318168 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.318137 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bdf36597-79ac-47ed-ba77-edcd447fdc53-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-qfh95\" (UID: \"bdf36597-79ac-47ed-ba77-edcd447fdc53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" Jul 29 04:09:01.319670 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.319645 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdf36597-79ac-47ed-ba77-edcd447fdc53-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-qfh95\" (UID: \"bdf36597-79ac-47ed-ba77-edcd447fdc53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" Jul 29 04:09:01.319767 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.319685 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bdf36597-79ac-47ed-ba77-edcd447fdc53-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-qfh95\" (UID: \"bdf36597-79ac-47ed-ba77-edcd447fdc53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" Jul 29 04:09:01.319767 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.319758 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bdf36597-79ac-47ed-ba77-edcd447fdc53-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-qfh95\" (UID: \"bdf36597-79ac-47ed-ba77-edcd447fdc53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" Jul 29 04:09:01.319838 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.319803 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bdf36597-79ac-47ed-ba77-edcd447fdc53-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-qfh95\" (UID: \"bdf36597-79ac-47ed-ba77-edcd447fdc53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" Jul 29 04:09:01.327946 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.327924 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdf36597-79ac-47ed-ba77-edcd447fdc53-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-qfh95\" (UID: \"bdf36597-79ac-47ed-ba77-edcd447fdc53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" Jul 29 04:09:01.328106 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.327973 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcfh5\" (UniqueName: \"kubernetes.io/projected/bdf36597-79ac-47ed-ba77-edcd447fdc53-kube-api-access-mcfh5\") pod \"istiod-openshift-gateway-7b7995d5cd-qfh95\" (UID: \"bdf36597-79ac-47ed-ba77-edcd447fdc53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" Jul 29 04:09:01.460459 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.460426 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" Jul 29 04:09:01.608943 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.608861 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95"] Jul 29 04:09:01.612399 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:09:01.612369 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf36597_79ac_47ed_ba77_edcd447fdc53.slice/crio-79a73dd7260ab778687032698b38e92b729c0723697d2124938b02b25aedbfa4 WatchSource:0}: Error finding container 79a73dd7260ab778687032698b38e92b729c0723697d2124938b02b25aedbfa4: Status 404 returned error can't find the container with id 79a73dd7260ab778687032698b38e92b729c0723697d2124938b02b25aedbfa4 Jul 29 04:09:01.871331 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.871243 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ndfqk" event={"ID":"0d1d3c77-b6f7-41c2-804c-cc829622aab0","Type":"ContainerStarted","Data":"377ee9a0822aec0c71d6d582b73d19fa8cce08ef998c4f589c562effbc0cdf1a"} Jul 29 04:09:01.872553 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.872524 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-76msd" event={"ID":"0ea18530-6257-4356-b3c7-bd18ed2f1e6c","Type":"ContainerStarted","Data":"6fd2f17c7472208ea2907981a8a3986bf8eacf67365edcdc2c6f60caa16c6989"} Jul 29 04:09:01.873936 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:01.873907 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" event={"ID":"bdf36597-79ac-47ed-ba77-edcd447fdc53","Type":"ContainerStarted","Data":"79a73dd7260ab778687032698b38e92b729c0723697d2124938b02b25aedbfa4"} Jul 29 04:09:05.689700 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:05.689660 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 04:09:05.690078 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:05.689740 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 04:09:05.892910 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:05.892811 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ndfqk" event={"ID":"0d1d3c77-b6f7-41c2-804c-cc829622aab0","Type":"ContainerStarted","Data":"21b8f065f3e04fb9ce0b9550e94b403632866c6be7e96b74df0fade63f3e52a8"} Jul 29 04:09:05.892910 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:05.892875 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-ndfqk" Jul 29 04:09:05.894310 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:05.894278 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-76msd" event={"ID":"0ea18530-6257-4356-b3c7-bd18ed2f1e6c","Type":"ContainerStarted","Data":"55bac6edc77a6614dd1633c17d8f97492def34a29f0c133d0ec297a9cd4f8807"} Jul 29 04:09:05.894438 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:05.894411 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-76msd" Jul 29 04:09:05.895801 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:05.895778 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-76ccb86b9-lh4m6" event={"ID":"df92f309-917b-4659-a2a8-6e9a1baa70f9","Type":"ContainerStarted","Data":"d275acb0a78b1b772486705d1a4125511a96a253d5faa121c40980d387f4e353"} Jul 29 04:09:05.897341 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:05.897322 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" event={"ID":"bdf36597-79ac-47ed-ba77-edcd447fdc53","Type":"ContainerStarted","Data":"8a804c0cca1ea05fba328574d57ff707b32c684c8a1991e45b8174e81fe9419b"} Jul 29 04:09:05.897466 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:05.897437 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" Jul 29 04:09:05.909782 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:05.909723 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-ndfqk" podStartSLOduration=1.488804465 podStartE2EDuration="5.909706738s" totalImagesPullingTime="4.420902273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:09:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:09:05.908717424 +0000 UTC m=+326.768092150" watchObservedRunningTime="2026-07-29 04:09:05.909706738 +0000 UTC m=+326.769081463" Jul 29 04:09:05.924077 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:05.924025 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-76ccb86b9-lh4m6" podStartSLOduration=2.161519053 podStartE2EDuration="6.924012009s" totalImagesPullingTime="4.762492956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:08:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:09:05.923068095 +0000 UTC m=+326.782442813" watchObservedRunningTime="2026-07-29 04:09:05.924012009 +0000 UTC m=+326.783386732" Jul 29 04:09:05.944473 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:05.944420 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-76msd" podStartSLOduration=2.180318627 podStartE2EDuration="6.944405465s" totalImagesPullingTime="4.764086838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:08:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:09:05.943360424 +0000 UTC m=+326.802735149" watchObservedRunningTime="2026-07-29 04:09:05.944405465 +0000 UTC m=+326.803780188" Jul 29 04:09:05.970124 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:05.970058 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" podStartSLOduration=0.895241994 podStartE2EDuration="4.970027874s" totalImagesPullingTime="4.07478588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:09:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:09:05.969142 +0000 UTC m=+326.828516725" watchObservedRunningTime="2026-07-29 04:09:05.970027874 +0000 UTC m=+326.829402599" Jul 29 04:09:06.904388 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:06.904305 2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-qfh95 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 04:09:06.904925 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:06.904899 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" podUID="bdf36597-79ac-47ed-ba77-edcd447fdc53" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 04:09:07.676235 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.676189 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw"] Jul 29 04:09:07.679760 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.679742 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:07.684314 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.684284 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-jdrfm\"" Jul 29 04:09:07.704033 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.704002 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw"] Jul 29 04:09:07.771309 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.771273 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fd035a-23fc-4221-a45d-19c7d74b5ae6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw\" (UID: \"63fd035a-23fc-4221-a45d-19c7d74b5ae6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:07.771466 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.771317 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63fd035a-23fc-4221-a45d-19c7d74b5ae6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw\" (UID: \"63fd035a-23fc-4221-a45d-19c7d74b5ae6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:07.771466 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.771339 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fd035a-23fc-4221-a45d-19c7d74b5ae6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw\" (UID: \"63fd035a-23fc-4221-a45d-19c7d74b5ae6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:07.771466 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.771426 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63fd035a-23fc-4221-a45d-19c7d74b5ae6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw\" (UID: \"63fd035a-23fc-4221-a45d-19c7d74b5ae6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:07.771567 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.771463 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63fd035a-23fc-4221-a45d-19c7d74b5ae6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw\" (UID: \"63fd035a-23fc-4221-a45d-19c7d74b5ae6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:07.771567 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.771503 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9r5c\" (UniqueName: \"kubernetes.io/projected/63fd035a-23fc-4221-a45d-19c7d74b5ae6-kube-api-access-d9r5c\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw\" (UID: \"63fd035a-23fc-4221-a45d-19c7d74b5ae6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:07.771567 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.771542 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63fd035a-23fc-4221-a45d-19c7d74b5ae6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw\" (UID: \"63fd035a-23fc-4221-a45d-19c7d74b5ae6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:07.771652 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.771622 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63fd035a-23fc-4221-a45d-19c7d74b5ae6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw\" (UID: \"63fd035a-23fc-4221-a45d-19c7d74b5ae6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:07.771681 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.771650 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63fd035a-23fc-4221-a45d-19c7d74b5ae6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw\" (UID: \"63fd035a-23fc-4221-a45d-19c7d74b5ae6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:07.872193 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.872150 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fd035a-23fc-4221-a45d-19c7d74b5ae6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw\" (UID: \"63fd035a-23fc-4221-a45d-19c7d74b5ae6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:07.872401 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.872234 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63fd035a-23fc-4221-a45d-19c7d74b5ae6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw\" (UID: \"63fd035a-23fc-4221-a45d-19c7d74b5ae6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:07.872401 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.872263 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fd035a-23fc-4221-a45d-19c7d74b5ae6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw\" (UID: \"63fd035a-23fc-4221-a45d-19c7d74b5ae6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:07.872401 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.872320 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63fd035a-23fc-4221-a45d-19c7d74b5ae6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw\" (UID: \"63fd035a-23fc-4221-a45d-19c7d74b5ae6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:07.872401 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.872350 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63fd035a-23fc-4221-a45d-19c7d74b5ae6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw\" (UID: \"63fd035a-23fc-4221-a45d-19c7d74b5ae6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:07.872614 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.872420 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9r5c\" (UniqueName: \"kubernetes.io/projected/63fd035a-23fc-4221-a45d-19c7d74b5ae6-kube-api-access-d9r5c\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw\" (UID: \"63fd035a-23fc-4221-a45d-19c7d74b5ae6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:07.872614 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.872477 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63fd035a-23fc-4221-a45d-19c7d74b5ae6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw\" (UID: \"63fd035a-23fc-4221-a45d-19c7d74b5ae6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:07.872614 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.872559 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63fd035a-23fc-4221-a45d-19c7d74b5ae6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw\" (UID: \"63fd035a-23fc-4221-a45d-19c7d74b5ae6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:07.872767 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.872626 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fd035a-23fc-4221-a45d-19c7d74b5ae6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw\" (UID: \"63fd035a-23fc-4221-a45d-19c7d74b5ae6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:07.872767 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.872677 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63fd035a-23fc-4221-a45d-19c7d74b5ae6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw\" (UID: \"63fd035a-23fc-4221-a45d-19c7d74b5ae6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:07.872860 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.872767 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fd035a-23fc-4221-a45d-19c7d74b5ae6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw\" (UID: \"63fd035a-23fc-4221-a45d-19c7d74b5ae6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:07.872860 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.872832 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63fd035a-23fc-4221-a45d-19c7d74b5ae6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw\" (UID: \"63fd035a-23fc-4221-a45d-19c7d74b5ae6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:07.872959 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.872909 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63fd035a-23fc-4221-a45d-19c7d74b5ae6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw\" (UID: \"63fd035a-23fc-4221-a45d-19c7d74b5ae6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:07.873057 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.873036 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63fd035a-23fc-4221-a45d-19c7d74b5ae6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw\" (UID: \"63fd035a-23fc-4221-a45d-19c7d74b5ae6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:07.874764 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.874741 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63fd035a-23fc-4221-a45d-19c7d74b5ae6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw\" (UID: \"63fd035a-23fc-4221-a45d-19c7d74b5ae6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:07.874992 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.874975 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63fd035a-23fc-4221-a45d-19c7d74b5ae6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw\" (UID: \"63fd035a-23fc-4221-a45d-19c7d74b5ae6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:07.897421 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.897396 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63fd035a-23fc-4221-a45d-19c7d74b5ae6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw\" (UID: \"63fd035a-23fc-4221-a45d-19c7d74b5ae6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:07.901607 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.901574 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9r5c\" (UniqueName: \"kubernetes.io/projected/63fd035a-23fc-4221-a45d-19c7d74b5ae6-kube-api-access-d9r5c\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw\" (UID: \"63fd035a-23fc-4221-a45d-19c7d74b5ae6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:07.989962 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:07.989921 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:08.129922 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:08.129888 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw"] Jul 29 04:09:08.133223 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:09:08.133183 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63fd035a_23fc_4221_a45d_19c7d74b5ae6.slice/crio-68ec8b32c2df3a564a666720d0813b8555c2e0aed78d3458c9c2478ddf097e9c WatchSource:0}: Error finding container 68ec8b32c2df3a564a666720d0813b8555c2e0aed78d3458c9c2478ddf097e9c: Status 404 returned error can't find the container with id 68ec8b32c2df3a564a666720d0813b8555c2e0aed78d3458c9c2478ddf097e9c Jul 29 04:09:08.913179 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:08.913146 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" event={"ID":"63fd035a-23fc-4221-a45d-19c7d74b5ae6","Type":"ContainerStarted","Data":"68ec8b32c2df3a564a666720d0813b8555c2e0aed78d3458c9c2478ddf097e9c"} Jul 29 04:09:09.659503 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:09.659468 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zxk52"] Jul 29 04:09:09.663848 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:09.663765 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxk52" Jul 29 04:09:09.668965 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:09.668937 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 29 04:09:09.669575 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:09.669555 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-8dw79\"" Jul 29 04:09:09.679310 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:09.678958 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zxk52"] Jul 29 04:09:09.791794 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:09.791758 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znfz9\" (UniqueName: \"kubernetes.io/projected/47950bb6-bfdb-4449-9079-088cb903ef41-kube-api-access-znfz9\") pod \"kserve-controller-manager-d6f4b65b8-zxk52\" (UID: \"47950bb6-bfdb-4449-9079-088cb903ef41\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxk52" Jul 29 04:09:09.791986 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:09.791816 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47950bb6-bfdb-4449-9079-088cb903ef41-cert\") pod \"kserve-controller-manager-d6f4b65b8-zxk52\" (UID: \"47950bb6-bfdb-4449-9079-088cb903ef41\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxk52" Jul 29 04:09:09.895242 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:09.895020 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znfz9\" (UniqueName: \"kubernetes.io/projected/47950bb6-bfdb-4449-9079-088cb903ef41-kube-api-access-znfz9\") pod \"kserve-controller-manager-d6f4b65b8-zxk52\" (UID: \"47950bb6-bfdb-4449-9079-088cb903ef41\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxk52" Jul 29 04:09:09.895242 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:09.895077 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47950bb6-bfdb-4449-9079-088cb903ef41-cert\") pod \"kserve-controller-manager-d6f4b65b8-zxk52\" (UID: \"47950bb6-bfdb-4449-9079-088cb903ef41\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxk52" Jul 29 04:09:09.898307 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:09.898236 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47950bb6-bfdb-4449-9079-088cb903ef41-cert\") pod \"kserve-controller-manager-d6f4b65b8-zxk52\" (UID: \"47950bb6-bfdb-4449-9079-088cb903ef41\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxk52" Jul 29 04:09:09.903745 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:09.903723 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" Jul 29 04:09:09.923433 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:09.923367 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znfz9\" (UniqueName: \"kubernetes.io/projected/47950bb6-bfdb-4449-9079-088cb903ef41-kube-api-access-znfz9\") pod \"kserve-controller-manager-d6f4b65b8-zxk52\" (UID: \"47950bb6-bfdb-4449-9079-088cb903ef41\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxk52" Jul 29 04:09:09.977953 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:09.977916 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxk52" Jul 29 04:09:10.854963 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:10.854926 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 04:09:10.855289 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:10.855004 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 04:09:10.855289 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:10.855046 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 04:09:10.922838 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:10.922808 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95"] Jul 29 04:09:10.923036 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:10.923013 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" podUID="bdf36597-79ac-47ed-ba77-edcd447fdc53" containerName="discovery" containerID="cri-o://8a804c0cca1ea05fba328574d57ff707b32c684c8a1991e45b8174e81fe9419b" gracePeriod=30 Jul 29 04:09:10.934772 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:10.934745 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zxk52"] Jul 29 04:09:10.937648 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:09:10.937619 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47950bb6_bfdb_4449_9079_088cb903ef41.slice/crio-9fd69f9dba2081f41e6a49e99e379336ce2aa20663e99e8c1fa2ca8bf772c7c4 WatchSource:0}: Error finding container 9fd69f9dba2081f41e6a49e99e379336ce2aa20663e99e8c1fa2ca8bf772c7c4: Status 404 returned error can't find the container with id 9fd69f9dba2081f41e6a49e99e379336ce2aa20663e99e8c1fa2ca8bf772c7c4 Jul 29 04:09:11.162062 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.162034 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" Jul 29 04:09:11.206654 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.206615 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdf36597-79ac-47ed-ba77-edcd447fdc53-local-certs\") pod \"bdf36597-79ac-47ed-ba77-edcd447fdc53\" (UID: \"bdf36597-79ac-47ed-ba77-edcd447fdc53\") " Jul 29 04:09:11.206837 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.206667 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bdf36597-79ac-47ed-ba77-edcd447fdc53-istio-csr-dns-cert\") pod \"bdf36597-79ac-47ed-ba77-edcd447fdc53\" (UID: \"bdf36597-79ac-47ed-ba77-edcd447fdc53\") " Jul 29 04:09:11.206837 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.206730 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bdf36597-79ac-47ed-ba77-edcd447fdc53-cacerts\") pod \"bdf36597-79ac-47ed-ba77-edcd447fdc53\" (UID: \"bdf36597-79ac-47ed-ba77-edcd447fdc53\") " Jul 29 04:09:11.206837 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.206750 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bdf36597-79ac-47ed-ba77-edcd447fdc53-istio-csr-ca-configmap\") pod \"bdf36597-79ac-47ed-ba77-edcd447fdc53\" (UID: \"bdf36597-79ac-47ed-ba77-edcd447fdc53\") " Jul 29 04:09:11.206837 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.206780 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcfh5\" (UniqueName: \"kubernetes.io/projected/bdf36597-79ac-47ed-ba77-edcd447fdc53-kube-api-access-mcfh5\") pod \"bdf36597-79ac-47ed-ba77-edcd447fdc53\" (UID: \"bdf36597-79ac-47ed-ba77-edcd447fdc53\") " Jul 29 04:09:11.207733 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.207083 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bdf36597-79ac-47ed-ba77-edcd447fdc53-istio-kubeconfig\") pod \"bdf36597-79ac-47ed-ba77-edcd447fdc53\" (UID: \"bdf36597-79ac-47ed-ba77-edcd447fdc53\") " Jul 29 04:09:11.207733 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.207141 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdf36597-79ac-47ed-ba77-edcd447fdc53-istio-token\") pod \"bdf36597-79ac-47ed-ba77-edcd447fdc53\" (UID: \"bdf36597-79ac-47ed-ba77-edcd447fdc53\") " Jul 29 04:09:11.207733 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.207372 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdf36597-79ac-47ed-ba77-edcd447fdc53-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "bdf36597-79ac-47ed-ba77-edcd447fdc53" (UID: "bdf36597-79ac-47ed-ba77-edcd447fdc53"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 04:09:11.207733 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.207459 2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bdf36597-79ac-47ed-ba77-edcd447fdc53-istio-csr-ca-configmap\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:09:11.209844 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.209803 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdf36597-79ac-47ed-ba77-edcd447fdc53-cacerts" (OuterVolumeSpecName: "cacerts") pod "bdf36597-79ac-47ed-ba77-edcd447fdc53" (UID: "bdf36597-79ac-47ed-ba77-edcd447fdc53"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 04:09:11.209844 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.209813 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdf36597-79ac-47ed-ba77-edcd447fdc53-kube-api-access-mcfh5" (OuterVolumeSpecName: "kube-api-access-mcfh5") pod "bdf36597-79ac-47ed-ba77-edcd447fdc53" (UID: "bdf36597-79ac-47ed-ba77-edcd447fdc53"). InnerVolumeSpecName "kube-api-access-mcfh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:09:11.210186 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.210157 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdf36597-79ac-47ed-ba77-edcd447fdc53-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "bdf36597-79ac-47ed-ba77-edcd447fdc53" (UID: "bdf36597-79ac-47ed-ba77-edcd447fdc53"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 04:09:11.210274 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.210191 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdf36597-79ac-47ed-ba77-edcd447fdc53-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "bdf36597-79ac-47ed-ba77-edcd447fdc53" (UID: "bdf36597-79ac-47ed-ba77-edcd447fdc53"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 04:09:11.210568 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.210533 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdf36597-79ac-47ed-ba77-edcd447fdc53-istio-token" (OuterVolumeSpecName: "istio-token") pod "bdf36597-79ac-47ed-ba77-edcd447fdc53" (UID: "bdf36597-79ac-47ed-ba77-edcd447fdc53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:09:11.211120 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.211102 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf36597-79ac-47ed-ba77-edcd447fdc53-local-certs" (OuterVolumeSpecName: "local-certs") pod "bdf36597-79ac-47ed-ba77-edcd447fdc53" (UID: "bdf36597-79ac-47ed-ba77-edcd447fdc53"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 04:09:11.307957 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.307926 2579 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdf36597-79ac-47ed-ba77-edcd447fdc53-local-certs\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:09:11.307957 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.307953 2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bdf36597-79ac-47ed-ba77-edcd447fdc53-istio-csr-dns-cert\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:09:11.307957 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.307964 2579 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bdf36597-79ac-47ed-ba77-edcd447fdc53-cacerts\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:09:11.308275 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.307973 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcfh5\" (UniqueName: \"kubernetes.io/projected/bdf36597-79ac-47ed-ba77-edcd447fdc53-kube-api-access-mcfh5\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:09:11.308275 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.307982 2579 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bdf36597-79ac-47ed-ba77-edcd447fdc53-istio-kubeconfig\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:09:11.308275 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.307990 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdf36597-79ac-47ed-ba77-edcd447fdc53-istio-token\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:09:11.928863 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.928826 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" event={"ID":"63fd035a-23fc-4221-a45d-19c7d74b5ae6","Type":"ContainerStarted","Data":"8642fdf4aaa25f90bba02f409308aa9d9e93022bb32b724adcbff855cd5c28d3"} Jul 29 04:09:11.930872 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.930846 2579 generic.go:356] "Generic (PLEG): container finished" podID="bdf36597-79ac-47ed-ba77-edcd447fdc53" containerID="8a804c0cca1ea05fba328574d57ff707b32c684c8a1991e45b8174e81fe9419b" exitCode=0 Jul 29 04:09:11.931021 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.930895 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" Jul 29 04:09:11.931021 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.930928 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" event={"ID":"bdf36597-79ac-47ed-ba77-edcd447fdc53","Type":"ContainerDied","Data":"8a804c0cca1ea05fba328574d57ff707b32c684c8a1991e45b8174e81fe9419b"} Jul 29 04:09:11.931021 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.930967 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95" event={"ID":"bdf36597-79ac-47ed-ba77-edcd447fdc53","Type":"ContainerDied","Data":"79a73dd7260ab778687032698b38e92b729c0723697d2124938b02b25aedbfa4"} Jul 29 04:09:11.931021 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.930995 2579 scope.go:117] "RemoveContainer" containerID="8a804c0cca1ea05fba328574d57ff707b32c684c8a1991e45b8174e81fe9419b" Jul 29 04:09:11.932864 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.932809 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxk52" event={"ID":"47950bb6-bfdb-4449-9079-088cb903ef41","Type":"ContainerStarted","Data":"9fd69f9dba2081f41e6a49e99e379336ce2aa20663e99e8c1fa2ca8bf772c7c4"} Jul 29 04:09:11.940968 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.940944 2579 scope.go:117] "RemoveContainer" containerID="8a804c0cca1ea05fba328574d57ff707b32c684c8a1991e45b8174e81fe9419b" Jul 29 04:09:11.941417 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:09:11.941393 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a804c0cca1ea05fba328574d57ff707b32c684c8a1991e45b8174e81fe9419b\": container with ID starting with 8a804c0cca1ea05fba328574d57ff707b32c684c8a1991e45b8174e81fe9419b not found: ID does not exist" containerID="8a804c0cca1ea05fba328574d57ff707b32c684c8a1991e45b8174e81fe9419b" Jul 29 04:09:11.941528 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.941428 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a804c0cca1ea05fba328574d57ff707b32c684c8a1991e45b8174e81fe9419b"} err="failed to get container status \"8a804c0cca1ea05fba328574d57ff707b32c684c8a1991e45b8174e81fe9419b\": rpc error: code = NotFound desc = could not find container \"8a804c0cca1ea05fba328574d57ff707b32c684c8a1991e45b8174e81fe9419b\": container with ID starting with 8a804c0cca1ea05fba328574d57ff707b32c684c8a1991e45b8174e81fe9419b not found: ID does not exist" Jul 29 04:09:11.952939 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.952890 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" podStartSLOduration=2.233499082 podStartE2EDuration="4.952868832s" totalImagesPullingTime="2.71936975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:09:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:09:11.951142199 +0000 UTC m=+332.810516949" watchObservedRunningTime="2026-07-29 04:09:11.952868832 +0000 UTC m=+332.812243557" Jul 29 04:09:11.966888 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.966851 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95"] Jul 29 04:09:11.970987 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.970960 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qfh95"] Jul 29 04:09:11.990497 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.990467 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:11.995749 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:11.995722 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:12.940032 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:12.939997 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:12.941249 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:12.941221 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw" Jul 29 04:09:13.590308 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:13.590229 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdf36597-79ac-47ed-ba77-edcd447fdc53" path="/var/lib/kubelet/pods/bdf36597-79ac-47ed-ba77-edcd447fdc53/volumes" Jul 29 04:09:13.945029 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:13.944994 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxk52" event={"ID":"47950bb6-bfdb-4449-9079-088cb903ef41","Type":"ContainerStarted","Data":"c3562de42b703da55910820d669873b5f7bf5aaf278d6447290e12dd872647e4"} Jul 29 04:09:13.945517 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:13.945196 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxk52" Jul 29 04:09:13.960831 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:13.960785 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxk52" podStartSLOduration=2.600046858 podStartE2EDuration="4.960771101s" totalImagesPullingTime="2.360724243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:09:13.960668479 +0000 UTC m=+334.820043203" watchObservedRunningTime="2026-07-29 04:09:13.960771101 +0000 UTC m=+334.820145824" Jul 29 04:09:16.906868 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:16.906839 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-ndfqk" Jul 29 04:09:16.926650 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:16.926624 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-76msd" Jul 29 04:09:44.954114 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:09:44.954081 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxk52" Jul 29 04:10:00.566109 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:00.566072 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zqcmn"] Jul 29 04:10:00.566497 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:00.566473 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdf36597-79ac-47ed-ba77-edcd447fdc53" containerName="discovery" Jul 29 04:10:00.566497 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:00.566488 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdf36597-79ac-47ed-ba77-edcd447fdc53" containerName="discovery" Jul 29 04:10:00.566588 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:00.566575 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdf36597-79ac-47ed-ba77-edcd447fdc53" containerName="discovery" Jul 29 04:10:00.581144 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:00.581112 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zqcmn"] Jul 29 04:10:00.581316 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:00.581242 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zqcmn" Jul 29 04:10:00.583574 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:00.583540 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 29 04:10:00.584509 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:00.584477 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-w5qv8\"" Jul 29 04:10:00.584678 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:00.584479 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 29 04:10:00.603350 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:00.603329 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qc7c\" (UniqueName: \"kubernetes.io/projected/05dac8a8-55f2-48f6-8c82-2d83cc7cc2c5-kube-api-access-7qc7c\") pod \"kuadrant-operator-catalog-zqcmn\" (UID: \"05dac8a8-55f2-48f6-8c82-2d83cc7cc2c5\") " pod="kuadrant-system/kuadrant-operator-catalog-zqcmn" Jul 29 04:10:00.704703 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:00.704669 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qc7c\" (UniqueName: \"kubernetes.io/projected/05dac8a8-55f2-48f6-8c82-2d83cc7cc2c5-kube-api-access-7qc7c\") pod \"kuadrant-operator-catalog-zqcmn\" (UID: \"05dac8a8-55f2-48f6-8c82-2d83cc7cc2c5\") " pod="kuadrant-system/kuadrant-operator-catalog-zqcmn" Jul 29 04:10:00.711857 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:00.711832 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qc7c\" (UniqueName: \"kubernetes.io/projected/05dac8a8-55f2-48f6-8c82-2d83cc7cc2c5-kube-api-access-7qc7c\") pod \"kuadrant-operator-catalog-zqcmn\" (UID: \"05dac8a8-55f2-48f6-8c82-2d83cc7cc2c5\") " pod="kuadrant-system/kuadrant-operator-catalog-zqcmn" Jul 29 04:10:00.891759 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:00.891661 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zqcmn" Jul 29 04:10:00.931485 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:00.931455 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zqcmn"] Jul 29 04:10:01.045823 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:01.045796 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zqcmn"] Jul 29 04:10:01.047755 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:10:01.047726 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05dac8a8_55f2_48f6_8c82_2d83cc7cc2c5.slice/crio-bfbff4fef0e511df0a8c27c785ac15e46ffb48329256ae12184e0caacbd4cf1e WatchSource:0}: Error finding container bfbff4fef0e511df0a8c27c785ac15e46ffb48329256ae12184e0caacbd4cf1e: Status 404 returned error can't find the container with id bfbff4fef0e511df0a8c27c785ac15e46ffb48329256ae12184e0caacbd4cf1e Jul 29 04:10:01.130547 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:01.130514 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zqcmn" event={"ID":"05dac8a8-55f2-48f6-8c82-2d83cc7cc2c5","Type":"ContainerStarted","Data":"bfbff4fef0e511df0a8c27c785ac15e46ffb48329256ae12184e0caacbd4cf1e"} Jul 29 04:10:01.142174 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:01.142103 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bmsrb"] Jul 29 04:10:01.146603 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:01.146587 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bmsrb" Jul 29 04:10:01.153757 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:01.153731 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bmsrb"] Jul 29 04:10:01.208929 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:01.208900 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzzfj\" (UniqueName: \"kubernetes.io/projected/f4b4a241-788d-4eef-b599-a4de941ddacb-kube-api-access-mzzfj\") pod \"kuadrant-operator-catalog-bmsrb\" (UID: \"f4b4a241-788d-4eef-b599-a4de941ddacb\") " pod="kuadrant-system/kuadrant-operator-catalog-bmsrb" Jul 29 04:10:01.309646 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:01.309611 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzzfj\" (UniqueName: \"kubernetes.io/projected/f4b4a241-788d-4eef-b599-a4de941ddacb-kube-api-access-mzzfj\") pod \"kuadrant-operator-catalog-bmsrb\" (UID: \"f4b4a241-788d-4eef-b599-a4de941ddacb\") " pod="kuadrant-system/kuadrant-operator-catalog-bmsrb" Jul 29 04:10:01.318642 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:01.318615 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzzfj\" (UniqueName: \"kubernetes.io/projected/f4b4a241-788d-4eef-b599-a4de941ddacb-kube-api-access-mzzfj\") pod \"kuadrant-operator-catalog-bmsrb\" (UID: \"f4b4a241-788d-4eef-b599-a4de941ddacb\") " pod="kuadrant-system/kuadrant-operator-catalog-bmsrb" Jul 29 04:10:01.456909 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:01.456865 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bmsrb" Jul 29 04:10:01.583312 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:01.583279 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bmsrb"] Jul 29 04:10:01.586758 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:10:01.586733 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b4a241_788d_4eef_b599_a4de941ddacb.slice/crio-4fceb62a53d3dff31b1a9cdaaedfe2ef1ef057645c249d41ab340bfb5578f00c WatchSource:0}: Error finding container 4fceb62a53d3dff31b1a9cdaaedfe2ef1ef057645c249d41ab340bfb5578f00c: Status 404 returned error can't find the container with id 4fceb62a53d3dff31b1a9cdaaedfe2ef1ef057645c249d41ab340bfb5578f00c Jul 29 04:10:02.135916 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:02.135879 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bmsrb" event={"ID":"f4b4a241-788d-4eef-b599-a4de941ddacb","Type":"ContainerStarted","Data":"4fceb62a53d3dff31b1a9cdaaedfe2ef1ef057645c249d41ab340bfb5578f00c"} Jul 29 04:10:04.146831 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:04.146723 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bmsrb" event={"ID":"f4b4a241-788d-4eef-b599-a4de941ddacb","Type":"ContainerStarted","Data":"30606b520638e8a1c127ed623d5a4daedc5e825fe6f3b1e8f9bea6f06bc00381"} Jul 29 04:10:04.148141 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:04.148112 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zqcmn" event={"ID":"05dac8a8-55f2-48f6-8c82-2d83cc7cc2c5","Type":"ContainerStarted","Data":"fdb0fc8549fd9bcfdc91d06c79fd14d4f960c2ba014c40e1bc72358ea64fc68e"} Jul 29 04:10:04.148280 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:04.148250 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-zqcmn" podUID="05dac8a8-55f2-48f6-8c82-2d83cc7cc2c5" containerName="registry-server" containerID="cri-o://fdb0fc8549fd9bcfdc91d06c79fd14d4f960c2ba014c40e1bc72358ea64fc68e" gracePeriod=2 Jul 29 04:10:04.162653 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:04.162598 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-bmsrb" podStartSLOduration=0.996356742 podStartE2EDuration="3.162580768s" totalImagesPullingTime="2.166224026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:10:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:10:04.161800306 +0000 UTC m=+385.021175030" watchObservedRunningTime="2026-07-29 04:10:04.162580768 +0000 UTC m=+385.021955493" Jul 29 04:10:04.175303 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:04.175264 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-zqcmn" podStartSLOduration=1.473196897 podStartE2EDuration="4.17524956s" totalImagesPullingTime="2.702052663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:10:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:10:04.174373823 +0000 UTC m=+385.033748547" watchObservedRunningTime="2026-07-29 04:10:04.17524956 +0000 UTC m=+385.034624283" Jul 29 04:10:04.390823 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:04.390798 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zqcmn" Jul 29 04:10:04.436543 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:04.436513 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qc7c\" (UniqueName: \"kubernetes.io/projected/05dac8a8-55f2-48f6-8c82-2d83cc7cc2c5-kube-api-access-7qc7c\") pod \"05dac8a8-55f2-48f6-8c82-2d83cc7cc2c5\" (UID: \"05dac8a8-55f2-48f6-8c82-2d83cc7cc2c5\") " Jul 29 04:10:04.438636 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:04.438607 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05dac8a8-55f2-48f6-8c82-2d83cc7cc2c5-kube-api-access-7qc7c" (OuterVolumeSpecName: "kube-api-access-7qc7c") pod "05dac8a8-55f2-48f6-8c82-2d83cc7cc2c5" (UID: "05dac8a8-55f2-48f6-8c82-2d83cc7cc2c5"). InnerVolumeSpecName "kube-api-access-7qc7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:10:04.537937 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:04.537905 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qc7c\" (UniqueName: \"kubernetes.io/projected/05dac8a8-55f2-48f6-8c82-2d83cc7cc2c5-kube-api-access-7qc7c\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:10:05.153648 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:05.153565 2579 generic.go:356] "Generic (PLEG): container finished" podID="05dac8a8-55f2-48f6-8c82-2d83cc7cc2c5" containerID="fdb0fc8549fd9bcfdc91d06c79fd14d4f960c2ba014c40e1bc72358ea64fc68e" exitCode=0 Jul 29 04:10:05.153648 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:05.153608 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zqcmn" Jul 29 04:10:05.154111 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:05.153652 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zqcmn" event={"ID":"05dac8a8-55f2-48f6-8c82-2d83cc7cc2c5","Type":"ContainerDied","Data":"fdb0fc8549fd9bcfdc91d06c79fd14d4f960c2ba014c40e1bc72358ea64fc68e"} Jul 29 04:10:05.154111 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:05.153698 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zqcmn" event={"ID":"05dac8a8-55f2-48f6-8c82-2d83cc7cc2c5","Type":"ContainerDied","Data":"bfbff4fef0e511df0a8c27c785ac15e46ffb48329256ae12184e0caacbd4cf1e"} Jul 29 04:10:05.154111 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:05.153724 2579 scope.go:117] "RemoveContainer" containerID="fdb0fc8549fd9bcfdc91d06c79fd14d4f960c2ba014c40e1bc72358ea64fc68e" Jul 29 04:10:05.162111 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:05.162094 2579 scope.go:117] "RemoveContainer" containerID="fdb0fc8549fd9bcfdc91d06c79fd14d4f960c2ba014c40e1bc72358ea64fc68e" Jul 29 04:10:05.162377 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:10:05.162357 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdb0fc8549fd9bcfdc91d06c79fd14d4f960c2ba014c40e1bc72358ea64fc68e\": container with ID starting with fdb0fc8549fd9bcfdc91d06c79fd14d4f960c2ba014c40e1bc72358ea64fc68e not found: ID does not exist" containerID="fdb0fc8549fd9bcfdc91d06c79fd14d4f960c2ba014c40e1bc72358ea64fc68e" Jul 29 04:10:05.162443 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:05.162388 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdb0fc8549fd9bcfdc91d06c79fd14d4f960c2ba014c40e1bc72358ea64fc68e"} err="failed to get container status \"fdb0fc8549fd9bcfdc91d06c79fd14d4f960c2ba014c40e1bc72358ea64fc68e\": rpc error: code = NotFound desc = could not find container \"fdb0fc8549fd9bcfdc91d06c79fd14d4f960c2ba014c40e1bc72358ea64fc68e\": container with ID starting with fdb0fc8549fd9bcfdc91d06c79fd14d4f960c2ba014c40e1bc72358ea64fc68e not found: ID does not exist" Jul 29 04:10:05.173402 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:05.173380 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zqcmn"] Jul 29 04:10:05.176454 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:05.176434 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zqcmn"] Jul 29 04:10:05.590849 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:05.590810 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05dac8a8-55f2-48f6-8c82-2d83cc7cc2c5" path="/var/lib/kubelet/pods/05dac8a8-55f2-48f6-8c82-2d83cc7cc2c5/volumes" Jul 29 04:10:11.457455 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:11.457396 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-bmsrb" Jul 29 04:10:11.457988 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:11.457499 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-bmsrb" Jul 29 04:10:11.479216 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:11.479176 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-bmsrb" Jul 29 04:10:12.203109 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:12.203084 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-bmsrb" Jul 29 04:10:32.400392 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:32.400357 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-fq588"] Jul 29 04:10:32.400912 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:32.400648 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05dac8a8-55f2-48f6-8c82-2d83cc7cc2c5" containerName="registry-server" Jul 29 04:10:32.400912 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:32.400659 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="05dac8a8-55f2-48f6-8c82-2d83cc7cc2c5" containerName="registry-server" Jul 29 04:10:32.400912 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:32.400707 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="05dac8a8-55f2-48f6-8c82-2d83cc7cc2c5" containerName="registry-server" Jul 29 04:10:32.403649 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:32.403629 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fq588" Jul 29 04:10:32.405903 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:32.405883 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 29 04:10:32.405903 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:32.405897 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-4j42c\"" Jul 29 04:10:32.414314 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:32.414285 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-fq588"] Jul 29 04:10:32.566068 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:32.566025 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm2bf\" (UniqueName: \"kubernetes.io/projected/f802ce80-3b32-4be6-bef3-7ea63ecb84f2-kube-api-access-cm2bf\") pod \"dns-operator-controller-manager-864dd78c4c-fq588\" (UID: \"f802ce80-3b32-4be6-bef3-7ea63ecb84f2\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fq588" Jul 29 04:10:32.666938 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:32.666820 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm2bf\" (UniqueName: \"kubernetes.io/projected/f802ce80-3b32-4be6-bef3-7ea63ecb84f2-kube-api-access-cm2bf\") pod \"dns-operator-controller-manager-864dd78c4c-fq588\" (UID: \"f802ce80-3b32-4be6-bef3-7ea63ecb84f2\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fq588" Jul 29 04:10:32.675503 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:32.675467 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm2bf\" (UniqueName: \"kubernetes.io/projected/f802ce80-3b32-4be6-bef3-7ea63ecb84f2-kube-api-access-cm2bf\") pod \"dns-operator-controller-manager-864dd78c4c-fq588\" (UID: \"f802ce80-3b32-4be6-bef3-7ea63ecb84f2\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fq588" Jul 29 04:10:32.713277 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:32.713241 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fq588" Jul 29 04:10:32.842014 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:32.841978 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-fq588"] Jul 29 04:10:32.844821 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:10:32.844788 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf802ce80_3b32_4be6_bef3_7ea63ecb84f2.slice/crio-3bc9319084e1b3a58af3811d216752bfbeea4b256467963421354f029226feb5 WatchSource:0}: Error finding container 3bc9319084e1b3a58af3811d216752bfbeea4b256467963421354f029226feb5: Status 404 returned error can't find the container with id 3bc9319084e1b3a58af3811d216752bfbeea4b256467963421354f029226feb5 Jul 29 04:10:33.265626 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:33.265585 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fq588" event={"ID":"f802ce80-3b32-4be6-bef3-7ea63ecb84f2","Type":"ContainerStarted","Data":"3bc9319084e1b3a58af3811d216752bfbeea4b256467963421354f029226feb5"} Jul 29 04:10:35.275858 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:35.275819 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fq588" event={"ID":"f802ce80-3b32-4be6-bef3-7ea63ecb84f2","Type":"ContainerStarted","Data":"ebc29117602c58d06badc43a4ce076a954c8fa2effea399910cc0ad1ee794adc"} Jul 29 04:10:35.276296 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:35.275939 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fq588" Jul 29 04:10:35.293162 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:35.293117 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fq588" podStartSLOduration=1.474780449 podStartE2EDuration="3.293103003s" totalImagesPullingTime="1.818322554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:10:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:10:35.292720027 +0000 UTC m=+416.152094850" watchObservedRunningTime="2026-07-29 04:10:35.293103003 +0000 UTC m=+416.152477727" Jul 29 04:10:35.843669 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:35.843633 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zdjj8"] Jul 29 04:10:35.846912 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:35.846895 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zdjj8" Jul 29 04:10:35.849113 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:35.849090 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-5hb6b\"" Jul 29 04:10:35.856521 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:35.856498 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zdjj8"] Jul 29 04:10:35.997484 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:35.997445 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfnv9\" (UniqueName: \"kubernetes.io/projected/c6b7be26-9837-4372-9b8e-17b30fff995a-kube-api-access-cfnv9\") pod \"limitador-operator-controller-manager-859d7b78bd-zdjj8\" (UID: \"c6b7be26-9837-4372-9b8e-17b30fff995a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zdjj8" Jul 29 04:10:36.097921 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:36.097821 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfnv9\" (UniqueName: \"kubernetes.io/projected/c6b7be26-9837-4372-9b8e-17b30fff995a-kube-api-access-cfnv9\") pod \"limitador-operator-controller-manager-859d7b78bd-zdjj8\" (UID: \"c6b7be26-9837-4372-9b8e-17b30fff995a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zdjj8" Jul 29 04:10:36.106160 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:36.106130 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfnv9\" (UniqueName: \"kubernetes.io/projected/c6b7be26-9837-4372-9b8e-17b30fff995a-kube-api-access-cfnv9\") pod \"limitador-operator-controller-manager-859d7b78bd-zdjj8\" (UID: \"c6b7be26-9837-4372-9b8e-17b30fff995a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zdjj8" Jul 29 04:10:36.157417 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:36.157381 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zdjj8" Jul 29 04:10:36.288976 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:36.288954 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zdjj8"] Jul 29 04:10:36.290822 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:10:36.290796 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6b7be26_9837_4372_9b8e_17b30fff995a.slice/crio-08a33a53ed8cde197bc4ba8b512a4415ef3fafbe195d91b23936fc94d885cdff WatchSource:0}: Error finding container 08a33a53ed8cde197bc4ba8b512a4415ef3fafbe195d91b23936fc94d885cdff: Status 404 returned error can't find the container with id 08a33a53ed8cde197bc4ba8b512a4415ef3fafbe195d91b23936fc94d885cdff Jul 29 04:10:37.288113 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:37.288073 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zdjj8" event={"ID":"c6b7be26-9837-4372-9b8e-17b30fff995a","Type":"ContainerStarted","Data":"08a33a53ed8cde197bc4ba8b512a4415ef3fafbe195d91b23936fc94d885cdff"} Jul 29 04:10:38.293936 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:38.293898 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zdjj8" event={"ID":"c6b7be26-9837-4372-9b8e-17b30fff995a","Type":"ContainerStarted","Data":"fc0c482c8b703314f199550a21fecefbee1973f33a91c0ec17322f6eac858051"} Jul 29 04:10:38.294519 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:38.294025 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zdjj8" Jul 29 04:10:38.309643 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:38.309556 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zdjj8" podStartSLOduration=1.545969769 podStartE2EDuration="3.3095403s" totalImagesPullingTime="1.763570531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:10:38.308666984 +0000 UTC m=+419.168041709" watchObservedRunningTime="2026-07-29 04:10:38.3095403 +0000 UTC m=+419.168915025" Jul 29 04:10:39.675689 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:39.675656 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-dsmlq"] Jul 29 04:10:39.681446 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:39.681429 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-dsmlq" Jul 29 04:10:39.683799 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:39.683779 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-hdgbt\"" Jul 29 04:10:39.688276 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:39.687956 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-dsmlq"] Jul 29 04:10:39.831120 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:39.831030 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5d8g\" (UniqueName: \"kubernetes.io/projected/fdd47a24-06a1-4569-a22e-c58f94bed861-kube-api-access-m5d8g\") pod \"authorino-operator-75cc8d5cb7-dsmlq\" (UID: \"fdd47a24-06a1-4569-a22e-c58f94bed861\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-dsmlq" Jul 29 04:10:39.931779 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:39.931742 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5d8g\" (UniqueName: \"kubernetes.io/projected/fdd47a24-06a1-4569-a22e-c58f94bed861-kube-api-access-m5d8g\") pod \"authorino-operator-75cc8d5cb7-dsmlq\" (UID: \"fdd47a24-06a1-4569-a22e-c58f94bed861\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-dsmlq" Jul 29 04:10:39.939387 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:39.939364 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5d8g\" (UniqueName: \"kubernetes.io/projected/fdd47a24-06a1-4569-a22e-c58f94bed861-kube-api-access-m5d8g\") pod \"authorino-operator-75cc8d5cb7-dsmlq\" (UID: \"fdd47a24-06a1-4569-a22e-c58f94bed861\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-dsmlq" Jul 29 04:10:39.991678 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:39.991650 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-dsmlq" Jul 29 04:10:40.130344 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:40.130318 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-dsmlq"] Jul 29 04:10:40.133063 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:10:40.133034 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd47a24_06a1_4569_a22e_c58f94bed861.slice/crio-3a2119a0d9d2a21d4d19cdd0621b4fe0a328d12b5c8df00b556bc117c6201f53 WatchSource:0}: Error finding container 3a2119a0d9d2a21d4d19cdd0621b4fe0a328d12b5c8df00b556bc117c6201f53: Status 404 returned error can't find the container with id 3a2119a0d9d2a21d4d19cdd0621b4fe0a328d12b5c8df00b556bc117c6201f53 Jul 29 04:10:40.303721 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:40.303676 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-dsmlq" event={"ID":"fdd47a24-06a1-4569-a22e-c58f94bed861","Type":"ContainerStarted","Data":"3a2119a0d9d2a21d4d19cdd0621b4fe0a328d12b5c8df00b556bc117c6201f53"} Jul 29 04:10:43.318990 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:43.318953 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-dsmlq" event={"ID":"fdd47a24-06a1-4569-a22e-c58f94bed861","Type":"ContainerStarted","Data":"28120ca5eb55a2bdc96ec0d73ed03264f79e54dc4e10f5314b50bea18d123069"} Jul 29 04:10:43.319464 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:43.319038 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-dsmlq" Jul 29 04:10:43.337488 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:43.337441 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-dsmlq" podStartSLOduration=2.159758102 podStartE2EDuration="4.33742497s" totalImagesPullingTime="2.177666868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:10:43.335770959 +0000 UTC m=+424.195145686" watchObservedRunningTime="2026-07-29 04:10:43.33742497 +0000 UTC m=+424.196799693" Jul 29 04:10:44.800044 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:44.800007 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g9ghz"] Jul 29 04:10:44.803432 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:44.803414 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g9ghz" Jul 29 04:10:44.805731 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:44.805712 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-nkp54\"" Jul 29 04:10:44.816683 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:44.816658 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g9ghz"] Jul 29 04:10:44.973398 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:44.973361 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e92a866c-4c7d-4bbe-abbb-5b580364037c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-g9ghz\" (UID: \"e92a866c-4c7d-4bbe-abbb-5b580364037c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g9ghz" Jul 29 04:10:44.973398 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:44.973404 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhnsk\" (UniqueName: \"kubernetes.io/projected/e92a866c-4c7d-4bbe-abbb-5b580364037c-kube-api-access-mhnsk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-g9ghz\" (UID: \"e92a866c-4c7d-4bbe-abbb-5b580364037c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g9ghz" Jul 29 04:10:45.073899 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:45.073800 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e92a866c-4c7d-4bbe-abbb-5b580364037c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-g9ghz\" (UID: \"e92a866c-4c7d-4bbe-abbb-5b580364037c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g9ghz" Jul 29 04:10:45.073899 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:45.073844 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhnsk\" (UniqueName: \"kubernetes.io/projected/e92a866c-4c7d-4bbe-abbb-5b580364037c-kube-api-access-mhnsk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-g9ghz\" (UID: \"e92a866c-4c7d-4bbe-abbb-5b580364037c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g9ghz" Jul 29 04:10:45.074247 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:45.074198 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e92a866c-4c7d-4bbe-abbb-5b580364037c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-g9ghz\" (UID: \"e92a866c-4c7d-4bbe-abbb-5b580364037c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g9ghz" Jul 29 04:10:45.083951 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:45.083930 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhnsk\" (UniqueName: \"kubernetes.io/projected/e92a866c-4c7d-4bbe-abbb-5b580364037c-kube-api-access-mhnsk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-g9ghz\" (UID: \"e92a866c-4c7d-4bbe-abbb-5b580364037c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g9ghz" Jul 29 04:10:45.113427 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:45.113396 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g9ghz" Jul 29 04:10:45.245313 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:45.245284 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g9ghz"] Jul 29 04:10:45.247961 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:10:45.247931 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92a866c_4c7d_4bbe_abbb_5b580364037c.slice/crio-93a3dd3117da59100f0e56c5bb2bbddc7e47b8a291b2aacae12f9b83c7d48fbc WatchSource:0}: Error finding container 93a3dd3117da59100f0e56c5bb2bbddc7e47b8a291b2aacae12f9b83c7d48fbc: Status 404 returned error can't find the container with id 93a3dd3117da59100f0e56c5bb2bbddc7e47b8a291b2aacae12f9b83c7d48fbc Jul 29 04:10:45.329277 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:45.329177 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g9ghz" event={"ID":"e92a866c-4c7d-4bbe-abbb-5b580364037c","Type":"ContainerStarted","Data":"93a3dd3117da59100f0e56c5bb2bbddc7e47b8a291b2aacae12f9b83c7d48fbc"} Jul 29 04:10:46.281921 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:46.281889 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fq588" Jul 29 04:10:49.300880 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:49.300848 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zdjj8" Jul 29 04:10:50.360319 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:50.360280 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g9ghz" event={"ID":"e92a866c-4c7d-4bbe-abbb-5b580364037c","Type":"ContainerStarted","Data":"5ddfcebfd7d0de343611d9277910142bc2c80759a401995f3eaefb01296726ff"} Jul 29 04:10:50.360720 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:50.360420 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g9ghz" Jul 29 04:10:50.377828 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:50.377778 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g9ghz" podStartSLOduration=1.404353047 podStartE2EDuration="6.377764326s" totalImagesPullingTime="4.973411279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:10:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:10:50.376618605 +0000 UTC m=+431.235993329" watchObservedRunningTime="2026-07-29 04:10:50.377764326 +0000 UTC m=+431.237139049" Jul 29 04:10:52.860774 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:52.860737 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-2jgl8"] Jul 29 04:10:52.864154 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:52.864136 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-2jgl8" Jul 29 04:10:52.866448 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:52.866420 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 29 04:10:52.866578 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:52.866426 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 29 04:10:52.866578 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:52.866566 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-g45cf\"" Jul 29 04:10:52.870975 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:52.870954 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-2jgl8"] Jul 29 04:10:52.926744 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:52.926710 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/709f3923-b331-49d9-9a0a-4cfec10c8137-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-2jgl8\" (UID: \"709f3923-b331-49d9-9a0a-4cfec10c8137\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-2jgl8" Jul 29 04:10:52.926910 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:52.926782 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x69n\" (UniqueName: \"kubernetes.io/projected/709f3923-b331-49d9-9a0a-4cfec10c8137-kube-api-access-6x69n\") pod \"kuadrant-console-plugin-66b5774985-2jgl8\" (UID: \"709f3923-b331-49d9-9a0a-4cfec10c8137\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-2jgl8" Jul 29 04:10:52.926910 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:52.926809 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/709f3923-b331-49d9-9a0a-4cfec10c8137-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-2jgl8\" (UID: \"709f3923-b331-49d9-9a0a-4cfec10c8137\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-2jgl8" Jul 29 04:10:53.027571 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:53.027535 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/709f3923-b331-49d9-9a0a-4cfec10c8137-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-2jgl8\" (UID: \"709f3923-b331-49d9-9a0a-4cfec10c8137\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-2jgl8" Jul 29 04:10:53.027738 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:53.027584 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/709f3923-b331-49d9-9a0a-4cfec10c8137-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-2jgl8\" (UID: \"709f3923-b331-49d9-9a0a-4cfec10c8137\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-2jgl8" Jul 29 04:10:53.027738 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:53.027640 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x69n\" (UniqueName: \"kubernetes.io/projected/709f3923-b331-49d9-9a0a-4cfec10c8137-kube-api-access-6x69n\") pod \"kuadrant-console-plugin-66b5774985-2jgl8\" (UID: \"709f3923-b331-49d9-9a0a-4cfec10c8137\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-2jgl8" Jul 29 04:10:53.028266 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:53.028243 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/709f3923-b331-49d9-9a0a-4cfec10c8137-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-2jgl8\" (UID: \"709f3923-b331-49d9-9a0a-4cfec10c8137\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-2jgl8" Jul 29 04:10:53.029947 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:53.029926 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/709f3923-b331-49d9-9a0a-4cfec10c8137-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-2jgl8\" (UID: \"709f3923-b331-49d9-9a0a-4cfec10c8137\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-2jgl8" Jul 29 04:10:53.035715 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:53.035686 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x69n\" (UniqueName: \"kubernetes.io/projected/709f3923-b331-49d9-9a0a-4cfec10c8137-kube-api-access-6x69n\") pod \"kuadrant-console-plugin-66b5774985-2jgl8\" (UID: \"709f3923-b331-49d9-9a0a-4cfec10c8137\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-2jgl8" Jul 29 04:10:53.174626 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:53.174530 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-2jgl8" Jul 29 04:10:53.300590 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:53.300430 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-2jgl8"] Jul 29 04:10:53.302862 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:10:53.302836 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod709f3923_b331_49d9_9a0a_4cfec10c8137.slice/crio-f0e00394fad4eac24a84db22fbc6ed0177d631950f93dce1f086abe8bf1e183c WatchSource:0}: Error finding container f0e00394fad4eac24a84db22fbc6ed0177d631950f93dce1f086abe8bf1e183c: Status 404 returned error can't find the container with id f0e00394fad4eac24a84db22fbc6ed0177d631950f93dce1f086abe8bf1e183c Jul 29 04:10:53.374482 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:53.374447 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-2jgl8" event={"ID":"709f3923-b331-49d9-9a0a-4cfec10c8137","Type":"ContainerStarted","Data":"f0e00394fad4eac24a84db22fbc6ed0177d631950f93dce1f086abe8bf1e183c"} Jul 29 04:10:54.327236 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:10:54.327187 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-dsmlq" Jul 29 04:11:01.367511 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:01.367469 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g9ghz" Jul 29 04:11:02.941180 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:02.941141 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g9ghz"] Jul 29 04:11:02.941661 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:02.941414 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g9ghz" podUID="e92a866c-4c7d-4bbe-abbb-5b580364037c" containerName="manager" containerID="cri-o://5ddfcebfd7d0de343611d9277910142bc2c80759a401995f3eaefb01296726ff" gracePeriod=2 Jul 29 04:11:02.952007 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:02.951972 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g9ghz"] Jul 29 04:11:02.960890 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:11:02.960837 2579 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"e92a866c-4c7d-4bbe-abbb-5b580364037c\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-g9ghz\": pods \"kuadrant-operator-controller-manager-8684c9ff49-g9ghz\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g9ghz" Jul 29 04:11:02.964904 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:02.964754 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ml9xx"] Jul 29 04:11:02.965282 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:02.965196 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e92a866c-4c7d-4bbe-abbb-5b580364037c" containerName="manager" Jul 29 04:11:02.965282 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:02.965232 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e92a866c-4c7d-4bbe-abbb-5b580364037c" containerName="manager" Jul 29 04:11:02.965457 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:02.965334 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e92a866c-4c7d-4bbe-abbb-5b580364037c" containerName="manager" Jul 29 04:11:02.970640 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:02.970617 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zdjj8"] Jul 29 04:11:02.970749 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:02.970732 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ml9xx" Jul 29 04:11:02.970971 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:02.970937 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zdjj8" podUID="c6b7be26-9837-4372-9b8e-17b30fff995a" containerName="manager" containerID="cri-o://fc0c482c8b703314f199550a21fecefbee1973f33a91c0ec17322f6eac858051" gracePeriod=2 Jul 29 04:11:02.976668 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:11:02.976556 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-g9ghz\" is forbidden: User \"system:node:ip-10-0-139-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-111.ec2.internal' and this object" podUID="e92a866c-4c7d-4bbe-abbb-5b580364037c" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g9ghz" Jul 29 04:11:02.986360 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:02.986194 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zdjj8"] Jul 29 04:11:02.990421 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:02.990361 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ml9xx"] Jul 29 04:11:02.998443 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:02.998416 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hppfm"] Jul 29 04:11:02.998894 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:02.998876 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6b7be26-9837-4372-9b8e-17b30fff995a" containerName="manager" Jul 29 04:11:02.998983 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:02.998898 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6b7be26-9837-4372-9b8e-17b30fff995a" containerName="manager" Jul 29 04:11:02.999038 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:02.998983 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6b7be26-9837-4372-9b8e-17b30fff995a" containerName="manager" Jul 29 04:11:03.001314 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:03.001288 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ml9xx\" (UID: \"7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ml9xx" Jul 29 04:11:03.001418 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:03.001326 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxnmf\" (UniqueName: \"kubernetes.io/projected/7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3-kube-api-access-fxnmf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ml9xx\" (UID: \"7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ml9xx" Jul 29 04:11:03.002266 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:03.002244 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hppfm" Jul 29 04:11:03.011288 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:11:03.011239 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-g9ghz\" is forbidden: User \"system:node:ip-10-0-139-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-111.ec2.internal' and this object" podUID="e92a866c-4c7d-4bbe-abbb-5b580364037c" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g9ghz" Jul 29 04:11:03.013249 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:11:03.013078 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zdjj8\" is forbidden: User \"system:node:ip-10-0-139-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-111.ec2.internal' and this object" podUID="c6b7be26-9837-4372-9b8e-17b30fff995a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zdjj8" Jul 29 04:11:03.015365 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:03.015090 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hppfm"] Jul 29 04:11:03.102086 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:03.102042 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ml9xx\" (UID: \"7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ml9xx" Jul 29 04:11:03.102257 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:03.102100 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxnmf\" (UniqueName: \"kubernetes.io/projected/7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3-kube-api-access-fxnmf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ml9xx\" (UID: \"7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ml9xx" Jul 29 04:11:03.102482 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:03.102461 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ml9xx\" (UID: \"7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ml9xx" Jul 29 04:11:03.110131 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:03.110105 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxnmf\" (UniqueName: \"kubernetes.io/projected/7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3-kube-api-access-fxnmf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ml9xx\" (UID: \"7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ml9xx" Jul 29 04:11:03.202816 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:03.202728 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hgtp\" (UniqueName: \"kubernetes.io/projected/95a2f8b7-667d-4cc9-aa41-171ed96d1d40-kube-api-access-8hgtp\") pod \"limitador-operator-controller-manager-859d7b78bd-hppfm\" (UID: \"95a2f8b7-667d-4cc9-aa41-171ed96d1d40\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hppfm" Jul 29 04:11:03.303554 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:03.303512 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hgtp\" (UniqueName: \"kubernetes.io/projected/95a2f8b7-667d-4cc9-aa41-171ed96d1d40-kube-api-access-8hgtp\") pod \"limitador-operator-controller-manager-859d7b78bd-hppfm\" (UID: \"95a2f8b7-667d-4cc9-aa41-171ed96d1d40\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hppfm" Jul 29 04:11:03.312144 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:03.312114 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hgtp\" (UniqueName: \"kubernetes.io/projected/95a2f8b7-667d-4cc9-aa41-171ed96d1d40-kube-api-access-8hgtp\") pod \"limitador-operator-controller-manager-859d7b78bd-hppfm\" (UID: \"95a2f8b7-667d-4cc9-aa41-171ed96d1d40\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hppfm" Jul 29 04:11:03.388076 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:03.388030 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ml9xx" Jul 29 04:11:03.394273 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:03.394239 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hppfm" Jul 29 04:11:15.297992 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.297964 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zdjj8" Jul 29 04:11:15.299951 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:11:15.299916 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zdjj8\" is forbidden: User \"system:node:ip-10-0-139-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-111.ec2.internal' and this object" podUID="c6b7be26-9837-4372-9b8e-17b30fff995a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zdjj8" Jul 29 04:11:15.306382 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.306361 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hppfm"] Jul 29 04:11:15.308489 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:11:15.308466 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a2f8b7_667d_4cc9_aa41_171ed96d1d40.slice/crio-ab1817008404b80a93515270bf7dc7081306b8fed5f873afa4b20ac7136e6360 WatchSource:0}: Error finding container ab1817008404b80a93515270bf7dc7081306b8fed5f873afa4b20ac7136e6360: Status 404 returned error can't find the container with id ab1817008404b80a93515270bf7dc7081306b8fed5f873afa4b20ac7136e6360 Jul 29 04:11:15.310945 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.310917 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g9ghz" Jul 29 04:11:15.313199 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:11:15.313174 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-g9ghz\" is forbidden: User \"system:node:ip-10-0-139-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-111.ec2.internal' and this object" podUID="e92a866c-4c7d-4bbe-abbb-5b580364037c" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g9ghz" Jul 29 04:11:15.314795 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:11:15.314768 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zdjj8\" is forbidden: User \"system:node:ip-10-0-139-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-111.ec2.internal' and this object" podUID="c6b7be26-9837-4372-9b8e-17b30fff995a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zdjj8" Jul 29 04:11:15.332386 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.332365 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ml9xx"] Jul 29 04:11:15.334575 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:11:15.334553 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f88794e_eba3_4b6d_aae2_c2bf5fd42bd3.slice/crio-0bfbb027c2ecb7407ab24b5dfcde3f3ccd4cc684422d67562b067d0150ea5be9 WatchSource:0}: Error finding container 0bfbb027c2ecb7407ab24b5dfcde3f3ccd4cc684422d67562b067d0150ea5be9: Status 404 returned error can't find the container with id 0bfbb027c2ecb7407ab24b5dfcde3f3ccd4cc684422d67562b067d0150ea5be9 Jul 29 04:11:15.411066 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.411010 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfnv9\" (UniqueName: \"kubernetes.io/projected/c6b7be26-9837-4372-9b8e-17b30fff995a-kube-api-access-cfnv9\") pod \"c6b7be26-9837-4372-9b8e-17b30fff995a\" (UID: \"c6b7be26-9837-4372-9b8e-17b30fff995a\") " Jul 29 04:11:15.411066 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.411048 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e92a866c-4c7d-4bbe-abbb-5b580364037c-extensions-socket-volume\") pod \"e92a866c-4c7d-4bbe-abbb-5b580364037c\" (UID: \"e92a866c-4c7d-4bbe-abbb-5b580364037c\") " Jul 29 04:11:15.411251 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.411226 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhnsk\" (UniqueName: \"kubernetes.io/projected/e92a866c-4c7d-4bbe-abbb-5b580364037c-kube-api-access-mhnsk\") pod \"e92a866c-4c7d-4bbe-abbb-5b580364037c\" (UID: \"e92a866c-4c7d-4bbe-abbb-5b580364037c\") " Jul 29 04:11:15.411508 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.411486 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92a866c-4c7d-4bbe-abbb-5b580364037c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e92a866c-4c7d-4bbe-abbb-5b580364037c" (UID: "e92a866c-4c7d-4bbe-abbb-5b580364037c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 04:11:15.426611 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.426580 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92a866c-4c7d-4bbe-abbb-5b580364037c-kube-api-access-mhnsk" (OuterVolumeSpecName: "kube-api-access-mhnsk") pod "e92a866c-4c7d-4bbe-abbb-5b580364037c" (UID: "e92a866c-4c7d-4bbe-abbb-5b580364037c"). InnerVolumeSpecName "kube-api-access-mhnsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:11:15.426720 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.426640 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6b7be26-9837-4372-9b8e-17b30fff995a-kube-api-access-cfnv9" (OuterVolumeSpecName: "kube-api-access-cfnv9") pod "c6b7be26-9837-4372-9b8e-17b30fff995a" (UID: "c6b7be26-9837-4372-9b8e-17b30fff995a"). InnerVolumeSpecName "kube-api-access-cfnv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:11:15.498307 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.498272 2579 generic.go:356] "Generic (PLEG): container finished" podID="e92a866c-4c7d-4bbe-abbb-5b580364037c" containerID="5ddfcebfd7d0de343611d9277910142bc2c80759a401995f3eaefb01296726ff" exitCode=0 Jul 29 04:11:15.498508 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.498318 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g9ghz" Jul 29 04:11:15.498508 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.498380 2579 scope.go:117] "RemoveContainer" containerID="5ddfcebfd7d0de343611d9277910142bc2c80759a401995f3eaefb01296726ff" Jul 29 04:11:15.500067 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.500040 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hppfm" event={"ID":"95a2f8b7-667d-4cc9-aa41-171ed96d1d40","Type":"ContainerStarted","Data":"0645d640544c0be7cc5f60394f785c6fec5c604f370981a440ca0783e924cd29"} Jul 29 04:11:15.500223 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.500088 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hppfm" event={"ID":"95a2f8b7-667d-4cc9-aa41-171ed96d1d40","Type":"ContainerStarted","Data":"ab1817008404b80a93515270bf7dc7081306b8fed5f873afa4b20ac7136e6360"} Jul 29 04:11:15.500223 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.500130 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hppfm" Jul 29 04:11:15.500708 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:11:15.500682 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-g9ghz\" is forbidden: User \"system:node:ip-10-0-139-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-111.ec2.internal' and this object" podUID="e92a866c-4c7d-4bbe-abbb-5b580364037c" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g9ghz" Jul 29 04:11:15.501885 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.501860 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-2jgl8" event={"ID":"709f3923-b331-49d9-9a0a-4cfec10c8137","Type":"ContainerStarted","Data":"475c439ea6682ea520f029619acd6dcaa395636271cf39817980f9d1e9b7949e"} Jul 29 04:11:15.502430 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:11:15.502392 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zdjj8\" is forbidden: User \"system:node:ip-10-0-139-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-111.ec2.internal' and this object" podUID="c6b7be26-9837-4372-9b8e-17b30fff995a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zdjj8" Jul 29 04:11:15.503174 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.503152 2579 generic.go:356] "Generic (PLEG): container finished" podID="c6b7be26-9837-4372-9b8e-17b30fff995a" containerID="fc0c482c8b703314f199550a21fecefbee1973f33a91c0ec17322f6eac858051" exitCode=0 Jul 29 04:11:15.503271 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.503181 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zdjj8" Jul 29 04:11:15.505049 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.505025 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ml9xx" event={"ID":"7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3","Type":"ContainerStarted","Data":"39302e705a3350931520faf1df1bf125b385fb30699d1e7b420de9abe86a3725"} Jul 29 04:11:15.505153 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.505071 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ml9xx" event={"ID":"7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3","Type":"ContainerStarted","Data":"0bfbb027c2ecb7407ab24b5dfcde3f3ccd4cc684422d67562b067d0150ea5be9"} Jul 29 04:11:15.505201 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.505156 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ml9xx" Jul 29 04:11:15.508601 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.508584 2579 scope.go:117] "RemoveContainer" containerID="5ddfcebfd7d0de343611d9277910142bc2c80759a401995f3eaefb01296726ff" Jul 29 04:11:15.508847 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:11:15.508829 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ddfcebfd7d0de343611d9277910142bc2c80759a401995f3eaefb01296726ff\": container with ID starting with 5ddfcebfd7d0de343611d9277910142bc2c80759a401995f3eaefb01296726ff not found: ID does not exist" containerID="5ddfcebfd7d0de343611d9277910142bc2c80759a401995f3eaefb01296726ff" Jul 29 04:11:15.508903 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.508856 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ddfcebfd7d0de343611d9277910142bc2c80759a401995f3eaefb01296726ff"} err="failed to get container status \"5ddfcebfd7d0de343611d9277910142bc2c80759a401995f3eaefb01296726ff\": rpc error: code = NotFound desc = could not find container \"5ddfcebfd7d0de343611d9277910142bc2c80759a401995f3eaefb01296726ff\": container with ID starting with 5ddfcebfd7d0de343611d9277910142bc2c80759a401995f3eaefb01296726ff not found: ID does not exist" Jul 29 04:11:15.508903 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.508875 2579 scope.go:117] "RemoveContainer" containerID="fc0c482c8b703314f199550a21fecefbee1973f33a91c0ec17322f6eac858051" Jul 29 04:11:15.512715 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.512693 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfnv9\" (UniqueName: \"kubernetes.io/projected/c6b7be26-9837-4372-9b8e-17b30fff995a-kube-api-access-cfnv9\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:11:15.512803 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.512719 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e92a866c-4c7d-4bbe-abbb-5b580364037c-extensions-socket-volume\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:11:15.512803 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.512733 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhnsk\" (UniqueName: \"kubernetes.io/projected/e92a866c-4c7d-4bbe-abbb-5b580364037c-kube-api-access-mhnsk\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:11:15.518228 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.518195 2579 scope.go:117] "RemoveContainer" containerID="fc0c482c8b703314f199550a21fecefbee1973f33a91c0ec17322f6eac858051" Jul 29 04:11:15.518493 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:11:15.518477 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc0c482c8b703314f199550a21fecefbee1973f33a91c0ec17322f6eac858051\": container with ID starting with fc0c482c8b703314f199550a21fecefbee1973f33a91c0ec17322f6eac858051 not found: ID does not exist" containerID="fc0c482c8b703314f199550a21fecefbee1973f33a91c0ec17322f6eac858051" Jul 29 04:11:15.518558 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.518500 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc0c482c8b703314f199550a21fecefbee1973f33a91c0ec17322f6eac858051"} err="failed to get container status \"fc0c482c8b703314f199550a21fecefbee1973f33a91c0ec17322f6eac858051\": rpc error: code = NotFound desc = could not find container \"fc0c482c8b703314f199550a21fecefbee1973f33a91c0ec17322f6eac858051\": container with ID starting with fc0c482c8b703314f199550a21fecefbee1973f33a91c0ec17322f6eac858051 not found: ID does not exist" Jul 29 04:11:15.520290 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.520247 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hppfm" podStartSLOduration=13.520234364 podStartE2EDuration="13.520234364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:11:15.51909441 +0000 UTC m=+456.378469137" watchObservedRunningTime="2026-07-29 04:11:15.520234364 +0000 UTC m=+456.379609066" Jul 29 04:11:15.533991 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.533947 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-2jgl8" podStartSLOduration=1.5772045110000001 podStartE2EDuration="23.533931283s" totalImagesPullingTime="21.956726772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:10:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:11:15.532571898 +0000 UTC m=+456.391946626" watchObservedRunningTime="2026-07-29 04:11:15.533931283 +0000 UTC m=+456.393306007" Jul 29 04:11:15.534536 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:11:15.534509 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-g9ghz\" is forbidden: User \"system:node:ip-10-0-139-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-111.ec2.internal' and this object" podUID="e92a866c-4c7d-4bbe-abbb-5b580364037c" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g9ghz" Jul 29 04:11:15.536003 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:11:15.535977 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zdjj8\" is forbidden: User \"system:node:ip-10-0-139-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-111.ec2.internal' and this object" podUID="c6b7be26-9837-4372-9b8e-17b30fff995a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zdjj8" Jul 29 04:11:15.538186 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:11:15.538158 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-g9ghz\" is forbidden: User \"system:node:ip-10-0-139-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-111.ec2.internal' and this object" podUID="e92a866c-4c7d-4bbe-abbb-5b580364037c" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g9ghz" Jul 29 04:11:15.556765 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.556723 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ml9xx" podStartSLOduration=13.556710401 podStartE2EDuration="13.556710401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:11:15.555446114 +0000 UTC m=+456.414820861" watchObservedRunningTime="2026-07-29 04:11:15.556710401 +0000 UTC m=+456.416085125" Jul 29 04:11:15.557185 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:11:15.557154 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zdjj8\" is forbidden: User \"system:node:ip-10-0-139-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-111.ec2.internal' and this object" podUID="c6b7be26-9837-4372-9b8e-17b30fff995a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zdjj8" Jul 29 04:11:15.590943 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.590904 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6b7be26-9837-4372-9b8e-17b30fff995a" path="/var/lib/kubelet/pods/c6b7be26-9837-4372-9b8e-17b30fff995a/volumes" Jul 29 04:11:15.591369 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:15.591348 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e92a866c-4c7d-4bbe-abbb-5b580364037c" path="/var/lib/kubelet/pods/e92a866c-4c7d-4bbe-abbb-5b580364037c/volumes" Jul 29 04:11:18.218878 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:18.218843 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ml9xx"] Jul 29 04:11:18.219295 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:18.219008 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ml9xx" podUID="7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3" containerName="manager" containerID="cri-o://39302e705a3350931520faf1df1bf125b385fb30699d1e7b420de9abe86a3725" gracePeriod=10 Jul 29 04:11:18.462150 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:18.462127 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ml9xx" Jul 29 04:11:18.523398 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:18.523318 2579 generic.go:356] "Generic (PLEG): container finished" podID="7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3" containerID="39302e705a3350931520faf1df1bf125b385fb30699d1e7b420de9abe86a3725" exitCode=0 Jul 29 04:11:18.523398 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:18.523371 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ml9xx" Jul 29 04:11:18.523581 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:18.523408 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ml9xx" event={"ID":"7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3","Type":"ContainerDied","Data":"39302e705a3350931520faf1df1bf125b385fb30699d1e7b420de9abe86a3725"} Jul 29 04:11:18.523581 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:18.523464 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ml9xx" event={"ID":"7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3","Type":"ContainerDied","Data":"0bfbb027c2ecb7407ab24b5dfcde3f3ccd4cc684422d67562b067d0150ea5be9"} Jul 29 04:11:18.523581 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:18.523488 2579 scope.go:117] "RemoveContainer" containerID="39302e705a3350931520faf1df1bf125b385fb30699d1e7b420de9abe86a3725" Jul 29 04:11:18.531832 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:18.531813 2579 scope.go:117] "RemoveContainer" containerID="39302e705a3350931520faf1df1bf125b385fb30699d1e7b420de9abe86a3725" Jul 29 04:11:18.532089 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:11:18.532069 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39302e705a3350931520faf1df1bf125b385fb30699d1e7b420de9abe86a3725\": container with ID starting with 39302e705a3350931520faf1df1bf125b385fb30699d1e7b420de9abe86a3725 not found: ID does not exist" containerID="39302e705a3350931520faf1df1bf125b385fb30699d1e7b420de9abe86a3725" Jul 29 04:11:18.532130 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:18.532098 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39302e705a3350931520faf1df1bf125b385fb30699d1e7b420de9abe86a3725"} err="failed to get container status \"39302e705a3350931520faf1df1bf125b385fb30699d1e7b420de9abe86a3725\": rpc error: code = NotFound desc = could not find container \"39302e705a3350931520faf1df1bf125b385fb30699d1e7b420de9abe86a3725\": container with ID starting with 39302e705a3350931520faf1df1bf125b385fb30699d1e7b420de9abe86a3725 not found: ID does not exist" Jul 29 04:11:18.534315 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:18.534297 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3-extensions-socket-volume\") pod \"7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3\" (UID: \"7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3\") " Jul 29 04:11:18.534399 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:18.534384 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxnmf\" (UniqueName: \"kubernetes.io/projected/7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3-kube-api-access-fxnmf\") pod \"7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3\" (UID: \"7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3\") " Jul 29 04:11:18.534641 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:18.534618 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3" (UID: "7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 04:11:18.536345 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:18.536323 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3-kube-api-access-fxnmf" (OuterVolumeSpecName: "kube-api-access-fxnmf") pod "7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3" (UID: "7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3"). InnerVolumeSpecName "kube-api-access-fxnmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:11:18.635257 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:18.635198 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3-extensions-socket-volume\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:11:18.635257 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:18.635250 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxnmf\" (UniqueName: \"kubernetes.io/projected/7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3-kube-api-access-fxnmf\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:11:18.845414 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:18.845378 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ml9xx"] Jul 29 04:11:18.853910 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:18.853880 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ml9xx"] Jul 29 04:11:19.591882 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:19.591848 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3" path="/var/lib/kubelet/pods/7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3/volumes" Jul 29 04:11:26.513903 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:26.513871 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hppfm" Jul 29 04:11:34.530724 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.530688 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs"] Jul 29 04:11:34.531292 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.531015 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3" containerName="manager" Jul 29 04:11:34.531292 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.531026 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3" containerName="manager" Jul 29 04:11:34.531292 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.531099 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f88794e-eba3-4b6d-aae2-c2bf5fd42bd3" containerName="manager" Jul 29 04:11:34.612181 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.612125 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs"] Jul 29 04:11:34.612378 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.612198 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:34.614432 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.614408 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-6j8qm\"" Jul 29 04:11:34.673455 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.673424 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f96d788-589f-4bf1-8873-92b1a8ed1b1c-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kjtrs\" (UID: \"2f96d788-589f-4bf1-8873-92b1a8ed1b1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:34.673648 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.673464 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f96d788-589f-4bf1-8873-92b1a8ed1b1c-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kjtrs\" (UID: \"2f96d788-589f-4bf1-8873-92b1a8ed1b1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:34.673648 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.673492 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f96d788-589f-4bf1-8873-92b1a8ed1b1c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kjtrs\" (UID: \"2f96d788-589f-4bf1-8873-92b1a8ed1b1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:34.673648 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.673526 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdtw5\" (UniqueName: \"kubernetes.io/projected/2f96d788-589f-4bf1-8873-92b1a8ed1b1c-kube-api-access-pdtw5\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kjtrs\" (UID: \"2f96d788-589f-4bf1-8873-92b1a8ed1b1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:34.673648 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.673555 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f96d788-589f-4bf1-8873-92b1a8ed1b1c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kjtrs\" (UID: \"2f96d788-589f-4bf1-8873-92b1a8ed1b1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:34.673648 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.673597 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f96d788-589f-4bf1-8873-92b1a8ed1b1c-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kjtrs\" (UID: \"2f96d788-589f-4bf1-8873-92b1a8ed1b1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:34.673648 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.673633 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f96d788-589f-4bf1-8873-92b1a8ed1b1c-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kjtrs\" (UID: \"2f96d788-589f-4bf1-8873-92b1a8ed1b1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:34.673865 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.673674 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f96d788-589f-4bf1-8873-92b1a8ed1b1c-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kjtrs\" (UID: \"2f96d788-589f-4bf1-8873-92b1a8ed1b1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:34.673865 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.673700 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f96d788-589f-4bf1-8873-92b1a8ed1b1c-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kjtrs\" (UID: \"2f96d788-589f-4bf1-8873-92b1a8ed1b1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:34.774506 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.774468 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdtw5\" (UniqueName: \"kubernetes.io/projected/2f96d788-589f-4bf1-8873-92b1a8ed1b1c-kube-api-access-pdtw5\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kjtrs\" (UID: \"2f96d788-589f-4bf1-8873-92b1a8ed1b1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:34.774506 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.774519 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f96d788-589f-4bf1-8873-92b1a8ed1b1c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kjtrs\" (UID: \"2f96d788-589f-4bf1-8873-92b1a8ed1b1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:34.774774 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.774544 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f96d788-589f-4bf1-8873-92b1a8ed1b1c-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kjtrs\" (UID: \"2f96d788-589f-4bf1-8873-92b1a8ed1b1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:34.774774 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.774576 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f96d788-589f-4bf1-8873-92b1a8ed1b1c-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kjtrs\" (UID: \"2f96d788-589f-4bf1-8873-92b1a8ed1b1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:34.774774 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.774625 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f96d788-589f-4bf1-8873-92b1a8ed1b1c-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kjtrs\" (UID: \"2f96d788-589f-4bf1-8873-92b1a8ed1b1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:34.774774 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.774653 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f96d788-589f-4bf1-8873-92b1a8ed1b1c-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kjtrs\" (UID: \"2f96d788-589f-4bf1-8873-92b1a8ed1b1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:34.774774 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.774718 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f96d788-589f-4bf1-8873-92b1a8ed1b1c-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kjtrs\" (UID: \"2f96d788-589f-4bf1-8873-92b1a8ed1b1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:34.774774 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.774748 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f96d788-589f-4bf1-8873-92b1a8ed1b1c-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kjtrs\" (UID: \"2f96d788-589f-4bf1-8873-92b1a8ed1b1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:34.774774 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.774771 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f96d788-589f-4bf1-8873-92b1a8ed1b1c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kjtrs\" (UID: \"2f96d788-589f-4bf1-8873-92b1a8ed1b1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:34.775103 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.775036 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f96d788-589f-4bf1-8873-92b1a8ed1b1c-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kjtrs\" (UID: \"2f96d788-589f-4bf1-8873-92b1a8ed1b1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:34.775103 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.775080 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f96d788-589f-4bf1-8873-92b1a8ed1b1c-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kjtrs\" (UID: \"2f96d788-589f-4bf1-8873-92b1a8ed1b1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:34.775229 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.775106 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f96d788-589f-4bf1-8873-92b1a8ed1b1c-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kjtrs\" (UID: \"2f96d788-589f-4bf1-8873-92b1a8ed1b1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:34.775344 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.775326 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f96d788-589f-4bf1-8873-92b1a8ed1b1c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kjtrs\" (UID: \"2f96d788-589f-4bf1-8873-92b1a8ed1b1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:34.775344 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.775337 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f96d788-589f-4bf1-8873-92b1a8ed1b1c-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kjtrs\" (UID: \"2f96d788-589f-4bf1-8873-92b1a8ed1b1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:34.777094 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.777067 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f96d788-589f-4bf1-8873-92b1a8ed1b1c-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kjtrs\" (UID: \"2f96d788-589f-4bf1-8873-92b1a8ed1b1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:34.777281 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.777265 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f96d788-589f-4bf1-8873-92b1a8ed1b1c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kjtrs\" (UID: \"2f96d788-589f-4bf1-8873-92b1a8ed1b1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:34.783274 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.782589 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdtw5\" (UniqueName: \"kubernetes.io/projected/2f96d788-589f-4bf1-8873-92b1a8ed1b1c-kube-api-access-pdtw5\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kjtrs\" (UID: \"2f96d788-589f-4bf1-8873-92b1a8ed1b1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:34.789781 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.789757 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f96d788-589f-4bf1-8873-92b1a8ed1b1c-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kjtrs\" (UID: \"2f96d788-589f-4bf1-8873-92b1a8ed1b1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:34.922321 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:34.922287 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:35.084816 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:35.084792 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs"] Jul 29 04:11:35.086704 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:11:35.086675 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f96d788_589f_4bf1_8873_92b1a8ed1b1c.slice/crio-bd14f6fbc2bee1ca982b5cfd1eef12a1fe4eccf50b61ab009a38c8437a71a825 WatchSource:0}: Error finding container bd14f6fbc2bee1ca982b5cfd1eef12a1fe4eccf50b61ab009a38c8437a71a825: Status 404 returned error can't find the container with id bd14f6fbc2bee1ca982b5cfd1eef12a1fe4eccf50b61ab009a38c8437a71a825 Jul 29 04:11:35.088922 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:35.088883 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 04:11:35.088994 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:35.088981 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 04:11:35.089032 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:35.089011 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 04:11:35.599111 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:35.599074 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" event={"ID":"2f96d788-589f-4bf1-8873-92b1a8ed1b1c","Type":"ContainerStarted","Data":"b076901cf42bb8f3fe1790c898851ada3c670428f9d010d98a7032875bd14153"} Jul 29 04:11:35.599518 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:35.599128 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" event={"ID":"2f96d788-589f-4bf1-8873-92b1a8ed1b1c","Type":"ContainerStarted","Data":"bd14f6fbc2bee1ca982b5cfd1eef12a1fe4eccf50b61ab009a38c8437a71a825"} Jul 29 04:11:35.622340 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:35.622292 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" podStartSLOduration=1.622274465 podStartE2EDuration="1.622274465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:11:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:11:35.620411677 +0000 UTC m=+476.479786412" watchObservedRunningTime="2026-07-29 04:11:35.622274465 +0000 UTC m=+476.481649189" Jul 29 04:11:35.923049 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:35.922964 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:36.927947 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:36.927914 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:37.607981 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:37.607952 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:37.608877 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:37.608857 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:11:50.479605 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:50.479571 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-q7m4q"] Jul 29 04:11:50.546911 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:50.546875 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-q7m4q"] Jul 29 04:11:50.547097 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:50.546953 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-q7m4q" Jul 29 04:11:50.549547 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:50.549296 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 29 04:11:50.580719 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:50.580683 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-q7m4q"] Jul 29 04:11:50.713255 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:50.713226 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0bcfd953-ce13-4b3b-80ae-31667dcd96d5-config-file\") pod \"limitador-limitador-64c94fd64f-q7m4q\" (UID: \"0bcfd953-ce13-4b3b-80ae-31667dcd96d5\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-q7m4q" Jul 29 04:11:50.713429 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:50.713340 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjhqb\" (UniqueName: \"kubernetes.io/projected/0bcfd953-ce13-4b3b-80ae-31667dcd96d5-kube-api-access-mjhqb\") pod \"limitador-limitador-64c94fd64f-q7m4q\" (UID: \"0bcfd953-ce13-4b3b-80ae-31667dcd96d5\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-q7m4q" Jul 29 04:11:50.814715 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:50.814627 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjhqb\" (UniqueName: \"kubernetes.io/projected/0bcfd953-ce13-4b3b-80ae-31667dcd96d5-kube-api-access-mjhqb\") pod \"limitador-limitador-64c94fd64f-q7m4q\" (UID: \"0bcfd953-ce13-4b3b-80ae-31667dcd96d5\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-q7m4q" Jul 29 04:11:50.814715 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:50.814668 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0bcfd953-ce13-4b3b-80ae-31667dcd96d5-config-file\") pod \"limitador-limitador-64c94fd64f-q7m4q\" (UID: \"0bcfd953-ce13-4b3b-80ae-31667dcd96d5\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-q7m4q" Jul 29 04:11:50.815424 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:50.815398 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0bcfd953-ce13-4b3b-80ae-31667dcd96d5-config-file\") pod \"limitador-limitador-64c94fd64f-q7m4q\" (UID: \"0bcfd953-ce13-4b3b-80ae-31667dcd96d5\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-q7m4q" Jul 29 04:11:50.821941 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:50.821919 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjhqb\" (UniqueName: \"kubernetes.io/projected/0bcfd953-ce13-4b3b-80ae-31667dcd96d5-kube-api-access-mjhqb\") pod \"limitador-limitador-64c94fd64f-q7m4q\" (UID: \"0bcfd953-ce13-4b3b-80ae-31667dcd96d5\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-q7m4q" Jul 29 04:11:50.860498 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:50.860463 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-q7m4q" Jul 29 04:11:50.992579 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:50.992547 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-q7m4q"] Jul 29 04:11:50.995904 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:11:50.995875 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bcfd953_ce13_4b3b_80ae_31667dcd96d5.slice/crio-9fa0875d20a179f8bd6f80157a6f1b4501e54390e6e2eedeb10baacecf33e29f WatchSource:0}: Error finding container 9fa0875d20a179f8bd6f80157a6f1b4501e54390e6e2eedeb10baacecf33e29f: Status 404 returned error can't find the container with id 9fa0875d20a179f8bd6f80157a6f1b4501e54390e6e2eedeb10baacecf33e29f Jul 29 04:11:51.253683 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:51.253651 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-sb48w"] Jul 29 04:11:51.258516 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:51.258494 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-sb48w" Jul 29 04:11:51.260944 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:51.260921 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cc2gt\"" Jul 29 04:11:51.267238 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:51.262892 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-sb48w"] Jul 29 04:11:51.418811 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:51.418774 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx8xx\" (UniqueName: \"kubernetes.io/projected/91ac6d97-e38f-45cb-b5ae-303faf77b4e8-kube-api-access-zx8xx\") pod \"authorino-58cd955675-sb48w\" (UID: \"91ac6d97-e38f-45cb-b5ae-303faf77b4e8\") " pod="kuadrant-system/authorino-58cd955675-sb48w" Jul 29 04:11:51.445237 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:51.445190 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-94blz"] Jul 29 04:11:51.448585 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:51.448567 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-94blz" Jul 29 04:11:51.454695 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:51.454670 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-94blz"] Jul 29 04:11:51.520197 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:51.520098 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx8xx\" (UniqueName: \"kubernetes.io/projected/91ac6d97-e38f-45cb-b5ae-303faf77b4e8-kube-api-access-zx8xx\") pod \"authorino-58cd955675-sb48w\" (UID: \"91ac6d97-e38f-45cb-b5ae-303faf77b4e8\") " pod="kuadrant-system/authorino-58cd955675-sb48w" Jul 29 04:11:51.527687 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:51.527661 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx8xx\" (UniqueName: \"kubernetes.io/projected/91ac6d97-e38f-45cb-b5ae-303faf77b4e8-kube-api-access-zx8xx\") pod \"authorino-58cd955675-sb48w\" (UID: \"91ac6d97-e38f-45cb-b5ae-303faf77b4e8\") " pod="kuadrant-system/authorino-58cd955675-sb48w" Jul 29 04:11:51.575180 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:51.575149 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-sb48w" Jul 29 04:11:51.621413 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:51.621357 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc4kt\" (UniqueName: \"kubernetes.io/projected/4d85e7c6-a890-4163-b073-90484403ad35-kube-api-access-mc4kt\") pod \"authorino-75d556bd77-94blz\" (UID: \"4d85e7c6-a890-4163-b073-90484403ad35\") " pod="kuadrant-system/authorino-75d556bd77-94blz" Jul 29 04:11:51.671572 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:51.671533 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-q7m4q" event={"ID":"0bcfd953-ce13-4b3b-80ae-31667dcd96d5","Type":"ContainerStarted","Data":"9fa0875d20a179f8bd6f80157a6f1b4501e54390e6e2eedeb10baacecf33e29f"} Jul 29 04:11:51.714162 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:51.714049 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-sb48w"] Jul 29 04:11:51.718956 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:11:51.716865 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91ac6d97_e38f_45cb_b5ae_303faf77b4e8.slice/crio-9d9f5d73beb3f896259a9a04475daabb0e2b25c7b4de66dcfdb8023e6122553e WatchSource:0}: Error finding container 9d9f5d73beb3f896259a9a04475daabb0e2b25c7b4de66dcfdb8023e6122553e: Status 404 returned error can't find the container with id 9d9f5d73beb3f896259a9a04475daabb0e2b25c7b4de66dcfdb8023e6122553e Jul 29 04:11:51.722585 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:51.722559 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc4kt\" (UniqueName: \"kubernetes.io/projected/4d85e7c6-a890-4163-b073-90484403ad35-kube-api-access-mc4kt\") pod \"authorino-75d556bd77-94blz\" (UID: \"4d85e7c6-a890-4163-b073-90484403ad35\") " pod="kuadrant-system/authorino-75d556bd77-94blz" Jul 29 04:11:51.732032 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:51.731980 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc4kt\" (UniqueName: \"kubernetes.io/projected/4d85e7c6-a890-4163-b073-90484403ad35-kube-api-access-mc4kt\") pod \"authorino-75d556bd77-94blz\" (UID: \"4d85e7c6-a890-4163-b073-90484403ad35\") " pod="kuadrant-system/authorino-75d556bd77-94blz" Jul 29 04:11:51.758440 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:51.758408 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-94blz" Jul 29 04:11:51.934593 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:51.933808 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-94blz"] Jul 29 04:11:51.936849 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:11:51.936818 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d85e7c6_a890_4163_b073_90484403ad35.slice/crio-466252da15bd2f65d164fa4c64ec22ac2c36c92163cac905bafba90d0632c8ec WatchSource:0}: Error finding container 466252da15bd2f65d164fa4c64ec22ac2c36c92163cac905bafba90d0632c8ec: Status 404 returned error can't find the container with id 466252da15bd2f65d164fa4c64ec22ac2c36c92163cac905bafba90d0632c8ec Jul 29 04:11:52.683661 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:52.683620 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-sb48w" event={"ID":"91ac6d97-e38f-45cb-b5ae-303faf77b4e8","Type":"ContainerStarted","Data":"9d9f5d73beb3f896259a9a04475daabb0e2b25c7b4de66dcfdb8023e6122553e"} Jul 29 04:11:52.686754 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:52.686653 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-94blz" event={"ID":"4d85e7c6-a890-4163-b073-90484403ad35","Type":"ContainerStarted","Data":"466252da15bd2f65d164fa4c64ec22ac2c36c92163cac905bafba90d0632c8ec"} Jul 29 04:11:56.712734 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:56.712697 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-q7m4q" event={"ID":"0bcfd953-ce13-4b3b-80ae-31667dcd96d5","Type":"ContainerStarted","Data":"7a0c95d2889746e37f89d426b343c82b7898414d91f7cc1ff97e3ba83dff0f07"} Jul 29 04:11:56.713228 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:56.712789 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-q7m4q" Jul 29 04:11:56.714260 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:56.714237 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-94blz" event={"ID":"4d85e7c6-a890-4163-b073-90484403ad35","Type":"ContainerStarted","Data":"62ce6874ad0e6d81882579d4a56175e9fc0488bed721222a6b3a21b16856b3ec"} Jul 29 04:11:56.715567 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:56.715532 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-sb48w" event={"ID":"91ac6d97-e38f-45cb-b5ae-303faf77b4e8","Type":"ContainerStarted","Data":"ceb36d55d79ff53837ee101a36ecc0d8a5bd83a33612cc4318d2da760af929ba"} Jul 29 04:11:56.729533 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:56.729494 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-q7m4q" podStartSLOduration=1.568544374 podStartE2EDuration="6.729480965s" totalImagesPullingTime="5.160936591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:11:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:11:56.727437135 +0000 UTC m=+497.586811861" watchObservedRunningTime="2026-07-29 04:11:56.729480965 +0000 UTC m=+497.588855698" Jul 29 04:11:56.740733 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:56.740694 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-94blz" podStartSLOduration=1.534390439 podStartE2EDuration="5.740678909s" totalImagesPullingTime="4.20628847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:11:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:11:56.739857097 +0000 UTC m=+497.599231822" watchObservedRunningTime="2026-07-29 04:11:56.740678909 +0000 UTC m=+497.600053633" Jul 29 04:11:56.753738 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:56.753695 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-sb48w" podStartSLOduration=1.32821453 podStartE2EDuration="5.753681556s" totalImagesPullingTime="4.425467026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:11:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:11:56.752422653 +0000 UTC m=+497.611797379" watchObservedRunningTime="2026-07-29 04:11:56.753681556 +0000 UTC m=+497.613056283" Jul 29 04:11:56.776346 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:56.776317 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-sb48w"] Jul 29 04:11:58.724223 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:58.724158 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-sb48w" podUID="91ac6d97-e38f-45cb-b5ae-303faf77b4e8" containerName="authorino" containerID="cri-o://ceb36d55d79ff53837ee101a36ecc0d8a5bd83a33612cc4318d2da760af929ba" gracePeriod=30 Jul 29 04:11:58.974014 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:58.973990 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-sb48w" Jul 29 04:11:58.984258 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:58.984189 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zx8xx\" (UniqueName: \"kubernetes.io/projected/91ac6d97-e38f-45cb-b5ae-303faf77b4e8-kube-api-access-zx8xx\") pod \"91ac6d97-e38f-45cb-b5ae-303faf77b4e8\" (UID: \"91ac6d97-e38f-45cb-b5ae-303faf77b4e8\") " Jul 29 04:11:58.986146 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:58.986122 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ac6d97-e38f-45cb-b5ae-303faf77b4e8-kube-api-access-zx8xx" (OuterVolumeSpecName: "kube-api-access-zx8xx") pod "91ac6d97-e38f-45cb-b5ae-303faf77b4e8" (UID: "91ac6d97-e38f-45cb-b5ae-303faf77b4e8"). InnerVolumeSpecName "kube-api-access-zx8xx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:11:59.085450 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:59.085416 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx8xx\" (UniqueName: \"kubernetes.io/projected/91ac6d97-e38f-45cb-b5ae-303faf77b4e8-kube-api-access-zx8xx\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:11:59.729733 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:59.729695 2579 generic.go:356] "Generic (PLEG): container finished" podID="91ac6d97-e38f-45cb-b5ae-303faf77b4e8" containerID="ceb36d55d79ff53837ee101a36ecc0d8a5bd83a33612cc4318d2da760af929ba" exitCode=0 Jul 29 04:11:59.730186 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:59.729739 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-sb48w" Jul 29 04:11:59.730186 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:59.729775 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-sb48w" event={"ID":"91ac6d97-e38f-45cb-b5ae-303faf77b4e8","Type":"ContainerDied","Data":"ceb36d55d79ff53837ee101a36ecc0d8a5bd83a33612cc4318d2da760af929ba"} Jul 29 04:11:59.730186 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:59.729826 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-sb48w" event={"ID":"91ac6d97-e38f-45cb-b5ae-303faf77b4e8","Type":"ContainerDied","Data":"9d9f5d73beb3f896259a9a04475daabb0e2b25c7b4de66dcfdb8023e6122553e"} Jul 29 04:11:59.730186 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:59.729843 2579 scope.go:117] "RemoveContainer" containerID="ceb36d55d79ff53837ee101a36ecc0d8a5bd83a33612cc4318d2da760af929ba" Jul 29 04:11:59.737893 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:59.737872 2579 scope.go:117] "RemoveContainer" containerID="ceb36d55d79ff53837ee101a36ecc0d8a5bd83a33612cc4318d2da760af929ba" Jul 29 04:11:59.738134 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:11:59.738115 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceb36d55d79ff53837ee101a36ecc0d8a5bd83a33612cc4318d2da760af929ba\": container with ID starting with ceb36d55d79ff53837ee101a36ecc0d8a5bd83a33612cc4318d2da760af929ba not found: ID does not exist" containerID="ceb36d55d79ff53837ee101a36ecc0d8a5bd83a33612cc4318d2da760af929ba" Jul 29 04:11:59.738199 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:59.738143 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceb36d55d79ff53837ee101a36ecc0d8a5bd83a33612cc4318d2da760af929ba"} err="failed to get container status \"ceb36d55d79ff53837ee101a36ecc0d8a5bd83a33612cc4318d2da760af929ba\": rpc error: code = NotFound desc = could not find container \"ceb36d55d79ff53837ee101a36ecc0d8a5bd83a33612cc4318d2da760af929ba\": container with ID starting with ceb36d55d79ff53837ee101a36ecc0d8a5bd83a33612cc4318d2da760af929ba not found: ID does not exist" Jul 29 04:11:59.744751 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:59.744727 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-sb48w"] Jul 29 04:11:59.748239 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:11:59.748219 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-sb48w"] Jul 29 04:12:01.591414 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:01.591377 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91ac6d97-e38f-45cb-b5ae-303faf77b4e8" path="/var/lib/kubelet/pods/91ac6d97-e38f-45cb-b5ae-303faf77b4e8/volumes" Jul 29 04:12:05.434555 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:05.434512 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-q7m4q"] Jul 29 04:12:05.435040 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:05.434817 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-q7m4q" podUID="0bcfd953-ce13-4b3b-80ae-31667dcd96d5" containerName="limitador" containerID="cri-o://7a0c95d2889746e37f89d426b343c82b7898414d91f7cc1ff97e3ba83dff0f07" gracePeriod=30 Jul 29 04:12:05.436980 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:05.436954 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-q7m4q" Jul 29 04:12:05.989500 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:05.989476 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-q7m4q" Jul 29 04:12:06.039493 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:06.039405 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjhqb\" (UniqueName: \"kubernetes.io/projected/0bcfd953-ce13-4b3b-80ae-31667dcd96d5-kube-api-access-mjhqb\") pod \"0bcfd953-ce13-4b3b-80ae-31667dcd96d5\" (UID: \"0bcfd953-ce13-4b3b-80ae-31667dcd96d5\") " Jul 29 04:12:06.039493 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:06.039453 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0bcfd953-ce13-4b3b-80ae-31667dcd96d5-config-file\") pod \"0bcfd953-ce13-4b3b-80ae-31667dcd96d5\" (UID: \"0bcfd953-ce13-4b3b-80ae-31667dcd96d5\") " Jul 29 04:12:06.039824 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:06.039801 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bcfd953-ce13-4b3b-80ae-31667dcd96d5-config-file" (OuterVolumeSpecName: "config-file") pod "0bcfd953-ce13-4b3b-80ae-31667dcd96d5" (UID: "0bcfd953-ce13-4b3b-80ae-31667dcd96d5"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 04:12:06.041473 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:06.041444 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bcfd953-ce13-4b3b-80ae-31667dcd96d5-kube-api-access-mjhqb" (OuterVolumeSpecName: "kube-api-access-mjhqb") pod "0bcfd953-ce13-4b3b-80ae-31667dcd96d5" (UID: "0bcfd953-ce13-4b3b-80ae-31667dcd96d5"). InnerVolumeSpecName "kube-api-access-mjhqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:12:06.141123 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:06.141089 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjhqb\" (UniqueName: \"kubernetes.io/projected/0bcfd953-ce13-4b3b-80ae-31667dcd96d5-kube-api-access-mjhqb\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:12:06.141123 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:06.141115 2579 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0bcfd953-ce13-4b3b-80ae-31667dcd96d5-config-file\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:12:06.761743 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:06.761712 2579 generic.go:356] "Generic (PLEG): container finished" podID="0bcfd953-ce13-4b3b-80ae-31667dcd96d5" containerID="7a0c95d2889746e37f89d426b343c82b7898414d91f7cc1ff97e3ba83dff0f07" exitCode=0 Jul 29 04:12:06.762309 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:06.761757 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-q7m4q" Jul 29 04:12:06.762309 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:06.761762 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-q7m4q" event={"ID":"0bcfd953-ce13-4b3b-80ae-31667dcd96d5","Type":"ContainerDied","Data":"7a0c95d2889746e37f89d426b343c82b7898414d91f7cc1ff97e3ba83dff0f07"} Jul 29 04:12:06.762309 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:06.761802 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-q7m4q" event={"ID":"0bcfd953-ce13-4b3b-80ae-31667dcd96d5","Type":"ContainerDied","Data":"9fa0875d20a179f8bd6f80157a6f1b4501e54390e6e2eedeb10baacecf33e29f"} Jul 29 04:12:06.762309 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:06.761824 2579 scope.go:117] "RemoveContainer" containerID="7a0c95d2889746e37f89d426b343c82b7898414d91f7cc1ff97e3ba83dff0f07" Jul 29 04:12:06.770373 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:06.770353 2579 scope.go:117] "RemoveContainer" containerID="7a0c95d2889746e37f89d426b343c82b7898414d91f7cc1ff97e3ba83dff0f07" Jul 29 04:12:06.770610 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:12:06.770592 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a0c95d2889746e37f89d426b343c82b7898414d91f7cc1ff97e3ba83dff0f07\": container with ID starting with 7a0c95d2889746e37f89d426b343c82b7898414d91f7cc1ff97e3ba83dff0f07 not found: ID does not exist" containerID="7a0c95d2889746e37f89d426b343c82b7898414d91f7cc1ff97e3ba83dff0f07" Jul 29 04:12:06.770670 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:06.770618 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a0c95d2889746e37f89d426b343c82b7898414d91f7cc1ff97e3ba83dff0f07"} err="failed to get container status \"7a0c95d2889746e37f89d426b343c82b7898414d91f7cc1ff97e3ba83dff0f07\": rpc error: code = NotFound desc = could not find container \"7a0c95d2889746e37f89d426b343c82b7898414d91f7cc1ff97e3ba83dff0f07\": container with ID starting with 7a0c95d2889746e37f89d426b343c82b7898414d91f7cc1ff97e3ba83dff0f07 not found: ID does not exist" Jul 29 04:12:06.781386 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:06.781360 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-q7m4q"] Jul 29 04:12:06.783487 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:06.783465 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-q7m4q"] Jul 29 04:12:07.592768 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:07.592731 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bcfd953-ce13-4b3b-80ae-31667dcd96d5" path="/var/lib/kubelet/pods/0bcfd953-ce13-4b3b-80ae-31667dcd96d5/volumes" Jul 29 04:12:08.428002 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:08.427970 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-9ll76"] Jul 29 04:12:08.428508 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:08.428346 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91ac6d97-e38f-45cb-b5ae-303faf77b4e8" containerName="authorino" Jul 29 04:12:08.428508 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:08.428359 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="91ac6d97-e38f-45cb-b5ae-303faf77b4e8" containerName="authorino" Jul 29 04:12:08.428508 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:08.428376 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bcfd953-ce13-4b3b-80ae-31667dcd96d5" containerName="limitador" Jul 29 04:12:08.428508 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:08.428382 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bcfd953-ce13-4b3b-80ae-31667dcd96d5" containerName="limitador" Jul 29 04:12:08.428508 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:08.428436 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bcfd953-ce13-4b3b-80ae-31667dcd96d5" containerName="limitador" Jul 29 04:12:08.428508 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:08.428443 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="91ac6d97-e38f-45cb-b5ae-303faf77b4e8" containerName="authorino" Jul 29 04:12:08.432548 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:08.432528 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9ll76" Jul 29 04:12:08.435201 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:08.435068 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-hv6tz\"" Jul 29 04:12:08.435201 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:08.435072 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 29 04:12:08.435201 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:08.435072 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 04:12:08.435862 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:08.435844 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 29 04:12:08.440739 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:08.440716 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-9ll76"] Jul 29 04:12:08.461556 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:08.461527 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pn4j\" (UniqueName: \"kubernetes.io/projected/a9918c69-9029-4a21-bbc7-86896a1c7aaa-kube-api-access-8pn4j\") pod \"postgres-69fb4594cf-9ll76\" (UID: \"a9918c69-9029-4a21-bbc7-86896a1c7aaa\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9ll76" Jul 29 04:12:08.461674 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:08.461570 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a9918c69-9029-4a21-bbc7-86896a1c7aaa-data\") pod \"postgres-69fb4594cf-9ll76\" (UID: \"a9918c69-9029-4a21-bbc7-86896a1c7aaa\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9ll76" Jul 29 04:12:08.562616 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:08.562582 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pn4j\" (UniqueName: \"kubernetes.io/projected/a9918c69-9029-4a21-bbc7-86896a1c7aaa-kube-api-access-8pn4j\") pod \"postgres-69fb4594cf-9ll76\" (UID: \"a9918c69-9029-4a21-bbc7-86896a1c7aaa\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9ll76" Jul 29 04:12:08.562795 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:08.562623 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a9918c69-9029-4a21-bbc7-86896a1c7aaa-data\") pod \"postgres-69fb4594cf-9ll76\" (UID: \"a9918c69-9029-4a21-bbc7-86896a1c7aaa\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9ll76" Jul 29 04:12:08.562982 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:08.562966 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a9918c69-9029-4a21-bbc7-86896a1c7aaa-data\") pod \"postgres-69fb4594cf-9ll76\" (UID: \"a9918c69-9029-4a21-bbc7-86896a1c7aaa\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9ll76" Jul 29 04:12:08.570679 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:08.570649 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pn4j\" (UniqueName: \"kubernetes.io/projected/a9918c69-9029-4a21-bbc7-86896a1c7aaa-kube-api-access-8pn4j\") pod \"postgres-69fb4594cf-9ll76\" (UID: \"a9918c69-9029-4a21-bbc7-86896a1c7aaa\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9ll76" Jul 29 04:12:08.744223 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:08.744129 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9ll76" Jul 29 04:12:08.872585 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:08.872450 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-9ll76"] Jul 29 04:12:08.874818 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:12:08.874788 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9918c69_9029_4a21_bbc7_86896a1c7aaa.slice/crio-300023757e37af98df07065db98bc3adda76b91f0c44852c01077560a3ef7c94 WatchSource:0}: Error finding container 300023757e37af98df07065db98bc3adda76b91f0c44852c01077560a3ef7c94: Status 404 returned error can't find the container with id 300023757e37af98df07065db98bc3adda76b91f0c44852c01077560a3ef7c94 Jul 29 04:12:09.776223 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:09.776163 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9ll76" event={"ID":"a9918c69-9029-4a21-bbc7-86896a1c7aaa","Type":"ContainerStarted","Data":"300023757e37af98df07065db98bc3adda76b91f0c44852c01077560a3ef7c94"} Jul 29 04:12:14.804169 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:14.804126 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9ll76" event={"ID":"a9918c69-9029-4a21-bbc7-86896a1c7aaa","Type":"ContainerStarted","Data":"e4c6803892e796fd1ea559ba802119e13e970e641fd3f5525fcc99e5e118d99f"} Jul 29 04:12:14.804573 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:14.804317 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9ll76" Jul 29 04:12:14.819162 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:14.819114 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9ll76" podStartSLOduration=1.653804449 podStartE2EDuration="6.819098988s" totalImagesPullingTime="5.165294539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:12:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:12:14.818167745 +0000 UTC m=+515.677542469" watchObservedRunningTime="2026-07-29 04:12:14.819098988 +0000 UTC m=+515.678473712" Jul 29 04:12:20.837764 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:20.837733 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9ll76" Jul 29 04:12:21.345055 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.345017 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-dbgq8"] Jul 29 04:12:21.351870 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.351844 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-dbgq8" Jul 29 04:12:21.354825 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.354797 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-dbgq8"] Jul 29 04:12:21.477531 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.477493 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2whhc\" (UniqueName: \"kubernetes.io/projected/bcda9dfa-b40c-4561-b7f2-527488766a60-kube-api-access-2whhc\") pod \"authorino-5d4b67c446-dbgq8\" (UID: \"bcda9dfa-b40c-4561-b7f2-527488766a60\") " pod="kuadrant-system/authorino-5d4b67c446-dbgq8" Jul 29 04:12:21.525683 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.525642 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-dbgq8"] Jul 29 04:12:21.525950 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:12:21.525927 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-2whhc], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-dbgq8" podUID="bcda9dfa-b40c-4561-b7f2-527488766a60" Jul 29 04:12:21.550933 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.550900 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-c6gw7"] Jul 29 04:12:21.554976 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.554958 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-c6gw7" Jul 29 04:12:21.557093 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.557073 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 29 04:12:21.563297 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.563276 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-c6gw7"] Jul 29 04:12:21.578653 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.578625 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2whhc\" (UniqueName: \"kubernetes.io/projected/bcda9dfa-b40c-4561-b7f2-527488766a60-kube-api-access-2whhc\") pod \"authorino-5d4b67c446-dbgq8\" (UID: \"bcda9dfa-b40c-4561-b7f2-527488766a60\") " pod="kuadrant-system/authorino-5d4b67c446-dbgq8" Jul 29 04:12:21.591931 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.591905 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2whhc\" (UniqueName: \"kubernetes.io/projected/bcda9dfa-b40c-4561-b7f2-527488766a60-kube-api-access-2whhc\") pod \"authorino-5d4b67c446-dbgq8\" (UID: \"bcda9dfa-b40c-4561-b7f2-527488766a60\") " pod="kuadrant-system/authorino-5d4b67c446-dbgq8" Jul 29 04:12:21.596021 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.595962 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-c6gw7"] Jul 29 04:12:21.596224 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:12:21.596184 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-qcqjc tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-qcqjc tls-cert]: context canceled" pod="kuadrant-system/authorino-76877994-c6gw7" podUID="15a0b1f1-e69e-403f-b64e-d1ccac37218f" Jul 29 04:12:21.623407 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.623378 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5999dc58c9-cg4bt"] Jul 29 04:12:21.627414 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.627394 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5999dc58c9-cg4bt" Jul 29 04:12:21.633491 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.633463 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5999dc58c9-cg4bt"] Jul 29 04:12:21.679518 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.679491 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/15a0b1f1-e69e-403f-b64e-d1ccac37218f-tls-cert\") pod \"authorino-76877994-c6gw7\" (UID: \"15a0b1f1-e69e-403f-b64e-d1ccac37218f\") " pod="kuadrant-system/authorino-76877994-c6gw7" Jul 29 04:12:21.679665 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.679545 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcqjc\" (UniqueName: \"kubernetes.io/projected/15a0b1f1-e69e-403f-b64e-d1ccac37218f-kube-api-access-qcqjc\") pod \"authorino-76877994-c6gw7\" (UID: \"15a0b1f1-e69e-403f-b64e-d1ccac37218f\") " pod="kuadrant-system/authorino-76877994-c6gw7" Jul 29 04:12:21.679665 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.679601 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6c678bb7-f13e-4f56-a64f-f719943fb1da-tls-cert\") pod \"authorino-5999dc58c9-cg4bt\" (UID: \"6c678bb7-f13e-4f56-a64f-f719943fb1da\") " pod="kuadrant-system/authorino-5999dc58c9-cg4bt" Jul 29 04:12:21.679740 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.679669 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtfmh\" (UniqueName: \"kubernetes.io/projected/6c678bb7-f13e-4f56-a64f-f719943fb1da-kube-api-access-mtfmh\") pod \"authorino-5999dc58c9-cg4bt\" (UID: \"6c678bb7-f13e-4f56-a64f-f719943fb1da\") " pod="kuadrant-system/authorino-5999dc58c9-cg4bt" Jul 29 04:12:21.780315 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.780285 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcqjc\" (UniqueName: \"kubernetes.io/projected/15a0b1f1-e69e-403f-b64e-d1ccac37218f-kube-api-access-qcqjc\") pod \"authorino-76877994-c6gw7\" (UID: \"15a0b1f1-e69e-403f-b64e-d1ccac37218f\") " pod="kuadrant-system/authorino-76877994-c6gw7" Jul 29 04:12:21.780450 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.780320 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6c678bb7-f13e-4f56-a64f-f719943fb1da-tls-cert\") pod \"authorino-5999dc58c9-cg4bt\" (UID: \"6c678bb7-f13e-4f56-a64f-f719943fb1da\") " pod="kuadrant-system/authorino-5999dc58c9-cg4bt" Jul 29 04:12:21.780450 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.780344 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtfmh\" (UniqueName: \"kubernetes.io/projected/6c678bb7-f13e-4f56-a64f-f719943fb1da-kube-api-access-mtfmh\") pod \"authorino-5999dc58c9-cg4bt\" (UID: \"6c678bb7-f13e-4f56-a64f-f719943fb1da\") " pod="kuadrant-system/authorino-5999dc58c9-cg4bt" Jul 29 04:12:21.780450 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.780427 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/15a0b1f1-e69e-403f-b64e-d1ccac37218f-tls-cert\") pod \"authorino-76877994-c6gw7\" (UID: \"15a0b1f1-e69e-403f-b64e-d1ccac37218f\") " pod="kuadrant-system/authorino-76877994-c6gw7" Jul 29 04:12:21.782942 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.782902 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6c678bb7-f13e-4f56-a64f-f719943fb1da-tls-cert\") pod \"authorino-5999dc58c9-cg4bt\" (UID: \"6c678bb7-f13e-4f56-a64f-f719943fb1da\") " pod="kuadrant-system/authorino-5999dc58c9-cg4bt" Jul 29 04:12:21.783065 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.782923 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/15a0b1f1-e69e-403f-b64e-d1ccac37218f-tls-cert\") pod \"authorino-76877994-c6gw7\" (UID: \"15a0b1f1-e69e-403f-b64e-d1ccac37218f\") " pod="kuadrant-system/authorino-76877994-c6gw7" Jul 29 04:12:21.787265 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.787237 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcqjc\" (UniqueName: \"kubernetes.io/projected/15a0b1f1-e69e-403f-b64e-d1ccac37218f-kube-api-access-qcqjc\") pod \"authorino-76877994-c6gw7\" (UID: \"15a0b1f1-e69e-403f-b64e-d1ccac37218f\") " pod="kuadrant-system/authorino-76877994-c6gw7" Jul 29 04:12:21.787368 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.787291 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtfmh\" (UniqueName: \"kubernetes.io/projected/6c678bb7-f13e-4f56-a64f-f719943fb1da-kube-api-access-mtfmh\") pod \"authorino-5999dc58c9-cg4bt\" (UID: \"6c678bb7-f13e-4f56-a64f-f719943fb1da\") " pod="kuadrant-system/authorino-5999dc58c9-cg4bt" Jul 29 04:12:21.834854 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.834828 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-dbgq8" Jul 29 04:12:21.835033 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.834827 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-c6gw7" Jul 29 04:12:21.842556 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.842536 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-dbgq8" Jul 29 04:12:21.845545 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.845526 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-c6gw7" Jul 29 04:12:21.937587 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.937559 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5999dc58c9-cg4bt" Jul 29 04:12:21.982302 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.982265 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2whhc\" (UniqueName: \"kubernetes.io/projected/bcda9dfa-b40c-4561-b7f2-527488766a60-kube-api-access-2whhc\") pod \"bcda9dfa-b40c-4561-b7f2-527488766a60\" (UID: \"bcda9dfa-b40c-4561-b7f2-527488766a60\") " Jul 29 04:12:21.982302 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.982296 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/15a0b1f1-e69e-403f-b64e-d1ccac37218f-tls-cert\") pod \"15a0b1f1-e69e-403f-b64e-d1ccac37218f\" (UID: \"15a0b1f1-e69e-403f-b64e-d1ccac37218f\") " Jul 29 04:12:21.982510 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.982322 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcqjc\" (UniqueName: \"kubernetes.io/projected/15a0b1f1-e69e-403f-b64e-d1ccac37218f-kube-api-access-qcqjc\") pod \"15a0b1f1-e69e-403f-b64e-d1ccac37218f\" (UID: \"15a0b1f1-e69e-403f-b64e-d1ccac37218f\") " Jul 29 04:12:21.984629 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.984586 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15a0b1f1-e69e-403f-b64e-d1ccac37218f-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "15a0b1f1-e69e-403f-b64e-d1ccac37218f" (UID: "15a0b1f1-e69e-403f-b64e-d1ccac37218f"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 04:12:21.984773 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.984623 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcda9dfa-b40c-4561-b7f2-527488766a60-kube-api-access-2whhc" (OuterVolumeSpecName: "kube-api-access-2whhc") pod "bcda9dfa-b40c-4561-b7f2-527488766a60" (UID: "bcda9dfa-b40c-4561-b7f2-527488766a60"). InnerVolumeSpecName "kube-api-access-2whhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:12:21.984773 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:21.984682 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a0b1f1-e69e-403f-b64e-d1ccac37218f-kube-api-access-qcqjc" (OuterVolumeSpecName: "kube-api-access-qcqjc") pod "15a0b1f1-e69e-403f-b64e-d1ccac37218f" (UID: "15a0b1f1-e69e-403f-b64e-d1ccac37218f"). InnerVolumeSpecName "kube-api-access-qcqjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:12:22.070299 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:22.070260 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5999dc58c9-cg4bt"] Jul 29 04:12:22.072520 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:12:22.072492 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c678bb7_f13e_4f56_a64f_f719943fb1da.slice/crio-2987641c2499c9dfeaa81ffa39abf66ba1b50f92fd8edc06d711aeadf50154e2 WatchSource:0}: Error finding container 2987641c2499c9dfeaa81ffa39abf66ba1b50f92fd8edc06d711aeadf50154e2: Status 404 returned error can't find the container with id 2987641c2499c9dfeaa81ffa39abf66ba1b50f92fd8edc06d711aeadf50154e2 Jul 29 04:12:22.083290 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:22.083265 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2whhc\" (UniqueName: \"kubernetes.io/projected/bcda9dfa-b40c-4561-b7f2-527488766a60-kube-api-access-2whhc\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:12:22.083290 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:22.083289 2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/15a0b1f1-e69e-403f-b64e-d1ccac37218f-tls-cert\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:12:22.083438 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:22.083299 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcqjc\" (UniqueName: \"kubernetes.io/projected/15a0b1f1-e69e-403f-b64e-d1ccac37218f-kube-api-access-qcqjc\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:12:22.840630 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:22.840536 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5999dc58c9-cg4bt" event={"ID":"6c678bb7-f13e-4f56-a64f-f719943fb1da","Type":"ContainerStarted","Data":"4d57913af1b814b8882741c13d35c6b33dcad6d3819e1b0438ed163e3965e702"} Jul 29 04:12:22.840630 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:22.840566 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-dbgq8" Jul 29 04:12:22.840630 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:22.840577 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-c6gw7" Jul 29 04:12:22.840630 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:22.840588 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5999dc58c9-cg4bt" event={"ID":"6c678bb7-f13e-4f56-a64f-f719943fb1da","Type":"ContainerStarted","Data":"2987641c2499c9dfeaa81ffa39abf66ba1b50f92fd8edc06d711aeadf50154e2"} Jul 29 04:12:22.858680 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:22.858628 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5999dc58c9-cg4bt" podStartSLOduration=1.455685171 podStartE2EDuration="1.858614553s" totalImagesPullingTime="402.929382ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:12:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:12:22.857089502 +0000 UTC m=+523.716464226" watchObservedRunningTime="2026-07-29 04:12:22.858614553 +0000 UTC m=+523.717989328" Jul 29 04:12:22.879506 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:22.879469 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-c6gw7"] Jul 29 04:12:22.885964 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:22.885937 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-c6gw7"] Jul 29 04:12:22.888399 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:22.888371 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-94blz"] Jul 29 04:12:22.888596 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:22.888573 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-94blz" podUID="4d85e7c6-a890-4163-b073-90484403ad35" containerName="authorino" containerID="cri-o://62ce6874ad0e6d81882579d4a56175e9fc0488bed721222a6b3a21b16856b3ec" gracePeriod=30 Jul 29 04:12:22.906217 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:22.906179 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-dbgq8"] Jul 29 04:12:22.911707 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:22.911681 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-dbgq8"] Jul 29 04:12:23.135860 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:23.135837 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-94blz" Jul 29 04:12:23.191860 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:23.191823 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mc4kt\" (UniqueName: \"kubernetes.io/projected/4d85e7c6-a890-4163-b073-90484403ad35-kube-api-access-mc4kt\") pod \"4d85e7c6-a890-4163-b073-90484403ad35\" (UID: \"4d85e7c6-a890-4163-b073-90484403ad35\") " Jul 29 04:12:23.193909 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:23.193880 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d85e7c6-a890-4163-b073-90484403ad35-kube-api-access-mc4kt" (OuterVolumeSpecName: "kube-api-access-mc4kt") pod "4d85e7c6-a890-4163-b073-90484403ad35" (UID: "4d85e7c6-a890-4163-b073-90484403ad35"). InnerVolumeSpecName "kube-api-access-mc4kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:12:23.293296 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:23.293260 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc4kt\" (UniqueName: \"kubernetes.io/projected/4d85e7c6-a890-4163-b073-90484403ad35-kube-api-access-mc4kt\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:12:23.591925 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:23.591890 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15a0b1f1-e69e-403f-b64e-d1ccac37218f" path="/var/lib/kubelet/pods/15a0b1f1-e69e-403f-b64e-d1ccac37218f/volumes" Jul 29 04:12:23.592138 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:23.592124 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcda9dfa-b40c-4561-b7f2-527488766a60" path="/var/lib/kubelet/pods/bcda9dfa-b40c-4561-b7f2-527488766a60/volumes" Jul 29 04:12:23.846426 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:23.846345 2579 generic.go:356] "Generic (PLEG): container finished" podID="4d85e7c6-a890-4163-b073-90484403ad35" containerID="62ce6874ad0e6d81882579d4a56175e9fc0488bed721222a6b3a21b16856b3ec" exitCode=0 Jul 29 04:12:23.846426 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:23.846388 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-94blz" Jul 29 04:12:23.846634 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:23.846430 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-94blz" event={"ID":"4d85e7c6-a890-4163-b073-90484403ad35","Type":"ContainerDied","Data":"62ce6874ad0e6d81882579d4a56175e9fc0488bed721222a6b3a21b16856b3ec"} Jul 29 04:12:23.846634 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:23.846475 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-94blz" event={"ID":"4d85e7c6-a890-4163-b073-90484403ad35","Type":"ContainerDied","Data":"466252da15bd2f65d164fa4c64ec22ac2c36c92163cac905bafba90d0632c8ec"} Jul 29 04:12:23.846634 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:23.846496 2579 scope.go:117] "RemoveContainer" containerID="62ce6874ad0e6d81882579d4a56175e9fc0488bed721222a6b3a21b16856b3ec" Jul 29 04:12:23.854810 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:23.854791 2579 scope.go:117] "RemoveContainer" containerID="62ce6874ad0e6d81882579d4a56175e9fc0488bed721222a6b3a21b16856b3ec" Jul 29 04:12:23.855070 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:12:23.855050 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62ce6874ad0e6d81882579d4a56175e9fc0488bed721222a6b3a21b16856b3ec\": container with ID starting with 62ce6874ad0e6d81882579d4a56175e9fc0488bed721222a6b3a21b16856b3ec not found: ID does not exist" containerID="62ce6874ad0e6d81882579d4a56175e9fc0488bed721222a6b3a21b16856b3ec" Jul 29 04:12:23.855120 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:23.855078 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62ce6874ad0e6d81882579d4a56175e9fc0488bed721222a6b3a21b16856b3ec"} err="failed to get container status \"62ce6874ad0e6d81882579d4a56175e9fc0488bed721222a6b3a21b16856b3ec\": rpc error: code = NotFound desc = could not find container \"62ce6874ad0e6d81882579d4a56175e9fc0488bed721222a6b3a21b16856b3ec\": container with ID starting with 62ce6874ad0e6d81882579d4a56175e9fc0488bed721222a6b3a21b16856b3ec not found: ID does not exist" Jul 29 04:12:23.860910 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:23.860886 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-94blz"] Jul 29 04:12:23.865004 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:23.864983 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-94blz"] Jul 29 04:12:25.592162 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:25.592129 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d85e7c6-a890-4163-b073-90484403ad35" path="/var/lib/kubelet/pods/4d85e7c6-a890-4163-b073-90484403ad35/volumes" Jul 29 04:12:27.408489 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:27.408455 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-59ffdf9686-h8wkp"] Jul 29 04:12:27.408939 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:27.408832 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d85e7c6-a890-4163-b073-90484403ad35" containerName="authorino" Jul 29 04:12:27.408939 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:27.408848 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d85e7c6-a890-4163-b073-90484403ad35" containerName="authorino" Jul 29 04:12:27.408939 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:27.408910 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d85e7c6-a890-4163-b073-90484403ad35" containerName="authorino" Jul 29 04:12:27.498099 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:27.498057 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59ffdf9686-h8wkp"] Jul 29 04:12:27.498307 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:27.498174 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59ffdf9686-h8wkp" Jul 29 04:12:27.500643 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:27.500611 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 29 04:12:27.500643 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:27.500635 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-45vj7\"" Jul 29 04:12:27.500838 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:27.500674 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 29 04:12:27.569181 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:27.569151 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b9ff676b5-ljc2v"] Jul 29 04:12:27.580760 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:27.580731 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b9ff676b5-ljc2v" Jul 29 04:12:27.580970 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:27.580933 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b9ff676b5-ljc2v"] Jul 29 04:12:27.631041 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:27.631011 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba3ac648-f9ca-4646-a433-e930d69d24f1-webhook-cert\") pod \"maas-controller-59ffdf9686-h8wkp\" (UID: \"ba3ac648-f9ca-4646-a433-e930d69d24f1\") " pod="opendatahub/maas-controller-59ffdf9686-h8wkp" Jul 29 04:12:27.631229 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:27.631052 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsw4g\" (UniqueName: \"kubernetes.io/projected/ba3ac648-f9ca-4646-a433-e930d69d24f1-kube-api-access-tsw4g\") pod \"maas-controller-59ffdf9686-h8wkp\" (UID: \"ba3ac648-f9ca-4646-a433-e930d69d24f1\") " pod="opendatahub/maas-controller-59ffdf9686-h8wkp" Jul 29 04:12:27.732250 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:27.732200 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8t6d\" (UniqueName: \"kubernetes.io/projected/235aea81-0258-4cb1-99a7-0258e6ed02df-kube-api-access-c8t6d\") pod \"maas-controller-7b9ff676b5-ljc2v\" (UID: \"235aea81-0258-4cb1-99a7-0258e6ed02df\") " pod="opendatahub/maas-controller-7b9ff676b5-ljc2v" Jul 29 04:12:27.732443 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:27.732262 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba3ac648-f9ca-4646-a433-e930d69d24f1-webhook-cert\") pod \"maas-controller-59ffdf9686-h8wkp\" (UID: \"ba3ac648-f9ca-4646-a433-e930d69d24f1\") " pod="opendatahub/maas-controller-59ffdf9686-h8wkp" Jul 29 04:12:27.732443 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:27.732329 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/235aea81-0258-4cb1-99a7-0258e6ed02df-webhook-cert\") pod \"maas-controller-7b9ff676b5-ljc2v\" (UID: \"235aea81-0258-4cb1-99a7-0258e6ed02df\") " pod="opendatahub/maas-controller-7b9ff676b5-ljc2v" Jul 29 04:12:27.732443 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:27.732378 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsw4g\" (UniqueName: \"kubernetes.io/projected/ba3ac648-f9ca-4646-a433-e930d69d24f1-kube-api-access-tsw4g\") pod \"maas-controller-59ffdf9686-h8wkp\" (UID: \"ba3ac648-f9ca-4646-a433-e930d69d24f1\") " pod="opendatahub/maas-controller-59ffdf9686-h8wkp" Jul 29 04:12:27.734653 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:27.734629 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba3ac648-f9ca-4646-a433-e930d69d24f1-webhook-cert\") pod \"maas-controller-59ffdf9686-h8wkp\" (UID: \"ba3ac648-f9ca-4646-a433-e930d69d24f1\") " pod="opendatahub/maas-controller-59ffdf9686-h8wkp" Jul 29 04:12:27.741345 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:27.741314 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsw4g\" (UniqueName: \"kubernetes.io/projected/ba3ac648-f9ca-4646-a433-e930d69d24f1-kube-api-access-tsw4g\") pod \"maas-controller-59ffdf9686-h8wkp\" (UID: \"ba3ac648-f9ca-4646-a433-e930d69d24f1\") " pod="opendatahub/maas-controller-59ffdf9686-h8wkp" Jul 29 04:12:27.808036 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:27.807998 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59ffdf9686-h8wkp" Jul 29 04:12:27.833243 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:27.833198 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8t6d\" (UniqueName: \"kubernetes.io/projected/235aea81-0258-4cb1-99a7-0258e6ed02df-kube-api-access-c8t6d\") pod \"maas-controller-7b9ff676b5-ljc2v\" (UID: \"235aea81-0258-4cb1-99a7-0258e6ed02df\") " pod="opendatahub/maas-controller-7b9ff676b5-ljc2v" Jul 29 04:12:27.833408 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:27.833263 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/235aea81-0258-4cb1-99a7-0258e6ed02df-webhook-cert\") pod \"maas-controller-7b9ff676b5-ljc2v\" (UID: \"235aea81-0258-4cb1-99a7-0258e6ed02df\") " pod="opendatahub/maas-controller-7b9ff676b5-ljc2v" Jul 29 04:12:27.835468 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:27.835448 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/235aea81-0258-4cb1-99a7-0258e6ed02df-webhook-cert\") pod \"maas-controller-7b9ff676b5-ljc2v\" (UID: \"235aea81-0258-4cb1-99a7-0258e6ed02df\") " pod="opendatahub/maas-controller-7b9ff676b5-ljc2v" Jul 29 04:12:27.841168 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:27.841141 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8t6d\" (UniqueName: \"kubernetes.io/projected/235aea81-0258-4cb1-99a7-0258e6ed02df-kube-api-access-c8t6d\") pod \"maas-controller-7b9ff676b5-ljc2v\" (UID: \"235aea81-0258-4cb1-99a7-0258e6ed02df\") " pod="opendatahub/maas-controller-7b9ff676b5-ljc2v" Jul 29 04:12:27.892591 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:27.892552 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b9ff676b5-ljc2v" Jul 29 04:12:27.982101 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:27.982072 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59ffdf9686-h8wkp"] Jul 29 04:12:27.985230 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:12:27.985186 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba3ac648_f9ca_4646_a433_e930d69d24f1.slice/crio-9d9e518b1d1ed92c1dd947714c92b9a32912a19f124d0dafa1ed09a7acd7b2e2 WatchSource:0}: Error finding container 9d9e518b1d1ed92c1dd947714c92b9a32912a19f124d0dafa1ed09a7acd7b2e2: Status 404 returned error can't find the container with id 9d9e518b1d1ed92c1dd947714c92b9a32912a19f124d0dafa1ed09a7acd7b2e2 Jul 29 04:12:28.040630 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:28.040607 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b9ff676b5-ljc2v"] Jul 29 04:12:28.042673 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:12:28.042644 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod235aea81_0258_4cb1_99a7_0258e6ed02df.slice/crio-f4e6ff416b22129aeed0eae2ee141bb8b9778f8bfad1026b5231e92dcdd4521c WatchSource:0}: Error finding container f4e6ff416b22129aeed0eae2ee141bb8b9778f8bfad1026b5231e92dcdd4521c: Status 404 returned error can't find the container with id f4e6ff416b22129aeed0eae2ee141bb8b9778f8bfad1026b5231e92dcdd4521c Jul 29 04:12:28.873138 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:28.873018 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b9ff676b5-ljc2v" event={"ID":"235aea81-0258-4cb1-99a7-0258e6ed02df","Type":"ContainerStarted","Data":"f4e6ff416b22129aeed0eae2ee141bb8b9778f8bfad1026b5231e92dcdd4521c"} Jul 29 04:12:28.875189 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:28.875149 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59ffdf9686-h8wkp" event={"ID":"ba3ac648-f9ca-4646-a433-e930d69d24f1","Type":"ContainerStarted","Data":"9d9e518b1d1ed92c1dd947714c92b9a32912a19f124d0dafa1ed09a7acd7b2e2"} Jul 29 04:12:31.891698 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:31.891599 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b9ff676b5-ljc2v" event={"ID":"235aea81-0258-4cb1-99a7-0258e6ed02df","Type":"ContainerStarted","Data":"e3fe0c97600c4616401b31c881725bcc16702b56e7c14199a75f0aa9010d70f1"} Jul 29 04:12:31.892139 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:31.891700 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b9ff676b5-ljc2v" Jul 29 04:12:31.893124 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:31.893101 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59ffdf9686-h8wkp" event={"ID":"ba3ac648-f9ca-4646-a433-e930d69d24f1","Type":"ContainerStarted","Data":"438edba3004597d879b81464adcaa91b4d20fa7322ca3f1e3f1fb957b779f846"} Jul 29 04:12:31.893269 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:31.893246 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-59ffdf9686-h8wkp" Jul 29 04:12:31.908690 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:31.908639 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7b9ff676b5-ljc2v" podStartSLOduration=2.129124081 podStartE2EDuration="4.90862301s" totalImagesPullingTime="2.779498929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:12:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:12:31.906809649 +0000 UTC m=+532.766184377" watchObservedRunningTime="2026-07-29 04:12:31.90862301 +0000 UTC m=+532.767997725" Jul 29 04:12:31.923782 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:31.923732 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-59ffdf9686-h8wkp" podStartSLOduration=2.094179201 podStartE2EDuration="4.9237177s" totalImagesPullingTime="2.829538499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:12:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:12:31.921950471 +0000 UTC m=+532.781325188" watchObservedRunningTime="2026-07-29 04:12:31.9237177 +0000 UTC m=+532.783092423" Jul 29 04:12:42.901190 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:42.901156 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b9ff676b5-ljc2v" Jul 29 04:12:42.901626 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:42.901261 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-59ffdf9686-h8wkp" Jul 29 04:12:42.958647 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:42.958606 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59ffdf9686-h8wkp"] Jul 29 04:12:42.958816 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:42.958792 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-59ffdf9686-h8wkp" podUID="ba3ac648-f9ca-4646-a433-e930d69d24f1" containerName="manager" containerID="cri-o://438edba3004597d879b81464adcaa91b4d20fa7322ca3f1e3f1fb957b779f846" gracePeriod=30 Jul 29 04:12:43.207056 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:43.207029 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59ffdf9686-h8wkp" Jul 29 04:12:43.372113 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:43.372079 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba3ac648-f9ca-4646-a433-e930d69d24f1-webhook-cert\") pod \"ba3ac648-f9ca-4646-a433-e930d69d24f1\" (UID: \"ba3ac648-f9ca-4646-a433-e930d69d24f1\") " Jul 29 04:12:43.372113 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:43.372117 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsw4g\" (UniqueName: \"kubernetes.io/projected/ba3ac648-f9ca-4646-a433-e930d69d24f1-kube-api-access-tsw4g\") pod \"ba3ac648-f9ca-4646-a433-e930d69d24f1\" (UID: \"ba3ac648-f9ca-4646-a433-e930d69d24f1\") " Jul 29 04:12:43.374363 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:43.374328 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3ac648-f9ca-4646-a433-e930d69d24f1-kube-api-access-tsw4g" (OuterVolumeSpecName: "kube-api-access-tsw4g") pod "ba3ac648-f9ca-4646-a433-e930d69d24f1" (UID: "ba3ac648-f9ca-4646-a433-e930d69d24f1"). InnerVolumeSpecName "kube-api-access-tsw4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:12:43.374488 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:43.374390 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba3ac648-f9ca-4646-a433-e930d69d24f1-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "ba3ac648-f9ca-4646-a433-e930d69d24f1" (UID: "ba3ac648-f9ca-4646-a433-e930d69d24f1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 04:12:43.473179 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:43.473096 2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba3ac648-f9ca-4646-a433-e930d69d24f1-webhook-cert\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:12:43.473179 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:43.473125 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsw4g\" (UniqueName: \"kubernetes.io/projected/ba3ac648-f9ca-4646-a433-e930d69d24f1-kube-api-access-tsw4g\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:12:43.950661 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:43.950625 2579 generic.go:356] "Generic (PLEG): container finished" podID="ba3ac648-f9ca-4646-a433-e930d69d24f1" containerID="438edba3004597d879b81464adcaa91b4d20fa7322ca3f1e3f1fb957b779f846" exitCode=0 Jul 29 04:12:43.951055 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:43.950666 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59ffdf9686-h8wkp" Jul 29 04:12:43.951055 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:43.950704 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59ffdf9686-h8wkp" event={"ID":"ba3ac648-f9ca-4646-a433-e930d69d24f1","Type":"ContainerDied","Data":"438edba3004597d879b81464adcaa91b4d20fa7322ca3f1e3f1fb957b779f846"} Jul 29 04:12:43.951055 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:43.950756 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59ffdf9686-h8wkp" event={"ID":"ba3ac648-f9ca-4646-a433-e930d69d24f1","Type":"ContainerDied","Data":"9d9e518b1d1ed92c1dd947714c92b9a32912a19f124d0dafa1ed09a7acd7b2e2"} Jul 29 04:12:43.951055 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:43.950779 2579 scope.go:117] "RemoveContainer" containerID="438edba3004597d879b81464adcaa91b4d20fa7322ca3f1e3f1fb957b779f846" Jul 29 04:12:43.958938 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:43.958922 2579 scope.go:117] "RemoveContainer" containerID="438edba3004597d879b81464adcaa91b4d20fa7322ca3f1e3f1fb957b779f846" Jul 29 04:12:43.959186 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:12:43.959168 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"438edba3004597d879b81464adcaa91b4d20fa7322ca3f1e3f1fb957b779f846\": container with ID starting with 438edba3004597d879b81464adcaa91b4d20fa7322ca3f1e3f1fb957b779f846 not found: ID does not exist" containerID="438edba3004597d879b81464adcaa91b4d20fa7322ca3f1e3f1fb957b779f846" Jul 29 04:12:43.959243 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:43.959195 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"438edba3004597d879b81464adcaa91b4d20fa7322ca3f1e3f1fb957b779f846"} err="failed to get container status \"438edba3004597d879b81464adcaa91b4d20fa7322ca3f1e3f1fb957b779f846\": rpc error: code = NotFound desc = could not find container \"438edba3004597d879b81464adcaa91b4d20fa7322ca3f1e3f1fb957b779f846\": container with ID starting with 438edba3004597d879b81464adcaa91b4d20fa7322ca3f1e3f1fb957b779f846 not found: ID does not exist" Jul 29 04:12:43.965682 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:43.965645 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59ffdf9686-h8wkp"] Jul 29 04:12:43.967140 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:43.967122 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-59ffdf9686-h8wkp"] Jul 29 04:12:45.591602 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:12:45.591569 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba3ac648-f9ca-4646-a433-e930d69d24f1" path="/var/lib/kubelet/pods/ba3ac648-f9ca-4646-a433-e930d69d24f1/volumes" Jul 29 04:13:03.950170 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:03.950136 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-97874b7d8-mc4q7"] Jul 29 04:13:03.950778 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:03.950556 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba3ac648-f9ca-4646-a433-e930d69d24f1" containerName="manager" Jul 29 04:13:03.950778 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:03.950571 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba3ac648-f9ca-4646-a433-e930d69d24f1" containerName="manager" Jul 29 04:13:03.950778 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:03.950666 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba3ac648-f9ca-4646-a433-e930d69d24f1" containerName="manager" Jul 29 04:13:03.953664 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:03.953646 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-97874b7d8-mc4q7" Jul 29 04:13:03.955862 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:03.955839 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 29 04:13:03.955969 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:03.955842 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rvzxd\"" Jul 29 04:13:03.962655 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:03.962629 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-97874b7d8-mc4q7"] Jul 29 04:13:04.040589 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:04.040548 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsf6l\" (UniqueName: \"kubernetes.io/projected/6500f16e-5496-4519-8b5b-354d30212cb2-kube-api-access-nsf6l\") pod \"maas-api-97874b7d8-mc4q7\" (UID: \"6500f16e-5496-4519-8b5b-354d30212cb2\") " pod="odh-ai-gateway-infra/maas-api-97874b7d8-mc4q7" Jul 29 04:13:04.040781 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:04.040604 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6500f16e-5496-4519-8b5b-354d30212cb2-openshift-service-ca\") pod \"maas-api-97874b7d8-mc4q7\" (UID: \"6500f16e-5496-4519-8b5b-354d30212cb2\") " pod="odh-ai-gateway-infra/maas-api-97874b7d8-mc4q7" Jul 29 04:13:04.040781 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:04.040650 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6500f16e-5496-4519-8b5b-354d30212cb2-maas-api-tls\") pod \"maas-api-97874b7d8-mc4q7\" (UID: \"6500f16e-5496-4519-8b5b-354d30212cb2\") " pod="odh-ai-gateway-infra/maas-api-97874b7d8-mc4q7" Jul 29 04:13:04.141419 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:04.141377 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsf6l\" (UniqueName: \"kubernetes.io/projected/6500f16e-5496-4519-8b5b-354d30212cb2-kube-api-access-nsf6l\") pod \"maas-api-97874b7d8-mc4q7\" (UID: \"6500f16e-5496-4519-8b5b-354d30212cb2\") " pod="odh-ai-gateway-infra/maas-api-97874b7d8-mc4q7" Jul 29 04:13:04.141419 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:04.141422 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6500f16e-5496-4519-8b5b-354d30212cb2-openshift-service-ca\") pod \"maas-api-97874b7d8-mc4q7\" (UID: \"6500f16e-5496-4519-8b5b-354d30212cb2\") " pod="odh-ai-gateway-infra/maas-api-97874b7d8-mc4q7" Jul 29 04:13:04.141676 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:04.141453 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6500f16e-5496-4519-8b5b-354d30212cb2-maas-api-tls\") pod \"maas-api-97874b7d8-mc4q7\" (UID: \"6500f16e-5496-4519-8b5b-354d30212cb2\") " pod="odh-ai-gateway-infra/maas-api-97874b7d8-mc4q7" Jul 29 04:13:04.141676 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:13:04.141541 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 29 04:13:04.141676 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:13:04.141611 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6500f16e-5496-4519-8b5b-354d30212cb2-maas-api-tls podName:6500f16e-5496-4519-8b5b-354d30212cb2 nodeName:}" failed. No retries permitted until 2026-07-29 04:13:04.641590016 +0000 UTC m=+565.500964730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6500f16e-5496-4519-8b5b-354d30212cb2-maas-api-tls") pod "maas-api-97874b7d8-mc4q7" (UID: "6500f16e-5496-4519-8b5b-354d30212cb2") : secret "maas-api-serving-cert" not found Jul 29 04:13:04.142125 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:04.142101 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6500f16e-5496-4519-8b5b-354d30212cb2-openshift-service-ca\") pod \"maas-api-97874b7d8-mc4q7\" (UID: \"6500f16e-5496-4519-8b5b-354d30212cb2\") " pod="odh-ai-gateway-infra/maas-api-97874b7d8-mc4q7" Jul 29 04:13:04.149304 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:04.149277 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsf6l\" (UniqueName: \"kubernetes.io/projected/6500f16e-5496-4519-8b5b-354d30212cb2-kube-api-access-nsf6l\") pod \"maas-api-97874b7d8-mc4q7\" (UID: \"6500f16e-5496-4519-8b5b-354d30212cb2\") " pod="odh-ai-gateway-infra/maas-api-97874b7d8-mc4q7" Jul 29 04:13:04.520656 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:04.520621 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-vmjvs"] Jul 29 04:13:04.524189 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:04.524162 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-vmjvs" Jul 29 04:13:04.526565 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:04.526545 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 29 04:13:04.526565 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:04.526543 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-p4gdw\"" Jul 29 04:13:04.535932 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:04.535902 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-vmjvs"] Jul 29 04:13:04.645155 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:04.645113 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6500f16e-5496-4519-8b5b-354d30212cb2-maas-api-tls\") pod \"maas-api-97874b7d8-mc4q7\" (UID: \"6500f16e-5496-4519-8b5b-354d30212cb2\") " pod="odh-ai-gateway-infra/maas-api-97874b7d8-mc4q7" Jul 29 04:13:04.645356 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:04.645174 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xqs4\" (UniqueName: \"kubernetes.io/projected/8d42dd32-a9fb-49a4-9c1e-34ba3e1a96c0-kube-api-access-5xqs4\") pod \"payload-processing-5cc6cd444b-vmjvs\" (UID: \"8d42dd32-a9fb-49a4-9c1e-34ba3e1a96c0\") " pod="openshift-ingress/payload-processing-5cc6cd444b-vmjvs" Jul 29 04:13:04.645540 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:04.645516 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d42dd32-a9fb-49a4-9c1e-34ba3e1a96c0-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-vmjvs\" (UID: \"8d42dd32-a9fb-49a4-9c1e-34ba3e1a96c0\") " pod="openshift-ingress/payload-processing-5cc6cd444b-vmjvs" Jul 29 04:13:04.647651 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:04.647632 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6500f16e-5496-4519-8b5b-354d30212cb2-maas-api-tls\") pod \"maas-api-97874b7d8-mc4q7\" (UID: \"6500f16e-5496-4519-8b5b-354d30212cb2\") " pod="odh-ai-gateway-infra/maas-api-97874b7d8-mc4q7" Jul 29 04:13:04.746738 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:04.746697 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xqs4\" (UniqueName: \"kubernetes.io/projected/8d42dd32-a9fb-49a4-9c1e-34ba3e1a96c0-kube-api-access-5xqs4\") pod \"payload-processing-5cc6cd444b-vmjvs\" (UID: \"8d42dd32-a9fb-49a4-9c1e-34ba3e1a96c0\") " pod="openshift-ingress/payload-processing-5cc6cd444b-vmjvs" Jul 29 04:13:04.746937 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:04.746762 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d42dd32-a9fb-49a4-9c1e-34ba3e1a96c0-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-vmjvs\" (UID: \"8d42dd32-a9fb-49a4-9c1e-34ba3e1a96c0\") " pod="openshift-ingress/payload-processing-5cc6cd444b-vmjvs" Jul 29 04:13:04.747499 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:04.747473 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d42dd32-a9fb-49a4-9c1e-34ba3e1a96c0-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-vmjvs\" (UID: \"8d42dd32-a9fb-49a4-9c1e-34ba3e1a96c0\") " pod="openshift-ingress/payload-processing-5cc6cd444b-vmjvs" Jul 29 04:13:04.767082 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:04.767060 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xqs4\" (UniqueName: \"kubernetes.io/projected/8d42dd32-a9fb-49a4-9c1e-34ba3e1a96c0-kube-api-access-5xqs4\") pod \"payload-processing-5cc6cd444b-vmjvs\" (UID: \"8d42dd32-a9fb-49a4-9c1e-34ba3e1a96c0\") " pod="openshift-ingress/payload-processing-5cc6cd444b-vmjvs" Jul 29 04:13:04.835994 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:04.835903 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-vmjvs" Jul 29 04:13:04.865533 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:04.865490 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-97874b7d8-mc4q7" Jul 29 04:13:04.983245 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:04.983140 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-vmjvs"] Jul 29 04:13:04.987412 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:13:04.987367 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d42dd32_a9fb_49a4_9c1e_34ba3e1a96c0.slice/crio-a8401e1b298f9572cf37be2f8b1bc4949627e55c5a2fb5a5fbc227380f5bb592 WatchSource:0}: Error finding container a8401e1b298f9572cf37be2f8b1bc4949627e55c5a2fb5a5fbc227380f5bb592: Status 404 returned error can't find the container with id a8401e1b298f9572cf37be2f8b1bc4949627e55c5a2fb5a5fbc227380f5bb592 Jul 29 04:13:05.014050 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:05.014032 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-97874b7d8-mc4q7"] Jul 29 04:13:05.016185 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:13:05.016158 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6500f16e_5496_4519_8b5b_354d30212cb2.slice/crio-d517700236d2d0c1e8b6951c1f6ca7b7f5f02dc0c8f2ed4eeefa6742c88417df WatchSource:0}: Error finding container d517700236d2d0c1e8b6951c1f6ca7b7f5f02dc0c8f2ed4eeefa6742c88417df: Status 404 returned error can't find the container with id d517700236d2d0c1e8b6951c1f6ca7b7f5f02dc0c8f2ed4eeefa6742c88417df Jul 29 04:13:05.050558 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:05.050515 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-97874b7d8-mc4q7" event={"ID":"6500f16e-5496-4519-8b5b-354d30212cb2","Type":"ContainerStarted","Data":"d517700236d2d0c1e8b6951c1f6ca7b7f5f02dc0c8f2ed4eeefa6742c88417df"} Jul 29 04:13:05.051608 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:05.051585 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-vmjvs" event={"ID":"8d42dd32-a9fb-49a4-9c1e-34ba3e1a96c0","Type":"ContainerStarted","Data":"a8401e1b298f9572cf37be2f8b1bc4949627e55c5a2fb5a5fbc227380f5bb592"} Jul 29 04:13:09.074595 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:09.074553 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-97874b7d8-mc4q7" event={"ID":"6500f16e-5496-4519-8b5b-354d30212cb2","Type":"ContainerStarted","Data":"8f96b2664b8664cd8879a37fe944a6a2319528d4ea7aa2a01ad1f1e40ac74b33"} Jul 29 04:13:09.075089 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:09.074733 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-97874b7d8-mc4q7" Jul 29 04:13:09.076010 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:09.075986 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-vmjvs" event={"ID":"8d42dd32-a9fb-49a4-9c1e-34ba3e1a96c0","Type":"ContainerStarted","Data":"e96d83a378f2f9aebd0546d5368c4096d5cf97b5a370d9a7e03b8fc7e0c1c2b7"} Jul 29 04:13:09.076137 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:09.076118 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-vmjvs" Jul 29 04:13:09.091173 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:09.091126 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-97874b7d8-mc4q7" podStartSLOduration=2.846357113 podStartE2EDuration="6.091111784s" totalImagesPullingTime="3.244754671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:13:09.089361941 +0000 UTC m=+569.948736664" watchObservedRunningTime="2026-07-29 04:13:09.091111784 +0000 UTC m=+569.950486508" Jul 29 04:13:09.103342 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:09.103302 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-vmjvs" podStartSLOduration=1.783670063 podStartE2EDuration="5.10328898s" totalImagesPullingTime="3.319618917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:13:09.102471334 +0000 UTC m=+569.961846094" watchObservedRunningTime="2026-07-29 04:13:09.10328898 +0000 UTC m=+569.962663704" Jul 29 04:13:12.091264 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:12.091236 2579 generic.go:356] "Generic (PLEG): container finished" podID="2f96d788-589f-4bf1-8873-92b1a8ed1b1c" containerID="b076901cf42bb8f3fe1790c898851ada3c670428f9d010d98a7032875bd14153" exitCode=0 Jul 29 04:13:12.091642 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:12.091304 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" event={"ID":"2f96d788-589f-4bf1-8873-92b1a8ed1b1c","Type":"ContainerDied","Data":"b076901cf42bb8f3fe1790c898851ada3c670428f9d010d98a7032875bd14153"} Jul 29 04:13:12.091685 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:12.091665 2579 scope.go:117] "RemoveContainer" containerID="b076901cf42bb8f3fe1790c898851ada3c670428f9d010d98a7032875bd14153" Jul 29 04:13:12.092600 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:12.092571 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 04:13:12.092671 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:12.092640 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 04:13:12.092708 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:12.092671 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 29 04:13:13.097661 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:13.097626 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" event={"ID":"2f96d788-589f-4bf1-8873-92b1a8ed1b1c","Type":"ContainerStarted","Data":"2d3b2a50e494dfbc5257b188882ccece77c31c15d41bdc609243e1b17d163552"} Jul 29 04:13:13.923357 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:13.923318 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:13:13.923536 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:13.923510 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-kjtrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Jul 29 04:13:13.923581 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:13.923560 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" podUID="2f96d788-589f-4bf1-8873-92b1a8ed1b1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Jul 29 04:13:14.923291 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:14.923255 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-kjtrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Jul 29 04:13:14.923684 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:14.923313 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" podUID="2f96d788-589f-4bf1-8873-92b1a8ed1b1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Jul 29 04:13:15.087063 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:15.087033 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-97874b7d8-mc4q7" Jul 29 04:13:15.923380 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:15.923341 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-kjtrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Jul 29 04:13:15.923781 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:15.923399 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" podUID="2f96d788-589f-4bf1-8873-92b1a8ed1b1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Jul 29 04:13:16.922571 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:16.922534 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-kjtrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Jul 29 04:13:16.922725 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:16.922590 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" podUID="2f96d788-589f-4bf1-8873-92b1a8ed1b1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Jul 29 04:13:17.927092 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:17.927055 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:13:17.927549 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:17.927527 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:13:17.928175 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:17.928156 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kjtrs" Jul 29 04:13:20.083632 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:20.083595 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-vmjvs" Jul 29 04:13:41.120276 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:41.120238 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-gxwxb"] Jul 29 04:13:41.127147 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:41.127123 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gxwxb" Jul 29 04:13:41.129518 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:41.129490 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 29 04:13:41.130199 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:41.130171 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-gxwxb"] Jul 29 04:13:41.130395 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:41.130377 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 29 04:13:41.130475 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:41.130401 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hrx5q\"" Jul 29 04:13:41.130475 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:41.130439 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 29 04:13:41.182379 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:41.182348 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4e78827-a544-47e4-ac4b-71fb42418fd8-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gxwxb\" (UID: \"e4e78827-a544-47e4-ac4b-71fb42418fd8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gxwxb" Jul 29 04:13:41.182543 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:41.182383 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4e78827-a544-47e4-ac4b-71fb42418fd8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gxwxb\" (UID: \"e4e78827-a544-47e4-ac4b-71fb42418fd8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gxwxb" Jul 29 04:13:41.182543 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:41.182413 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls26h\" (UniqueName: \"kubernetes.io/projected/e4e78827-a544-47e4-ac4b-71fb42418fd8-kube-api-access-ls26h\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gxwxb\" (UID: \"e4e78827-a544-47e4-ac4b-71fb42418fd8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gxwxb" Jul 29 04:13:41.182543 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:41.182501 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4e78827-a544-47e4-ac4b-71fb42418fd8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gxwxb\" (UID: \"e4e78827-a544-47e4-ac4b-71fb42418fd8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gxwxb" Jul 29 04:13:41.182543 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:41.182531 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e78827-a544-47e4-ac4b-71fb42418fd8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gxwxb\" (UID: \"e4e78827-a544-47e4-ac4b-71fb42418fd8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gxwxb" Jul 29 04:13:41.182689 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:41.182577 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4e78827-a544-47e4-ac4b-71fb42418fd8-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gxwxb\" (UID: \"e4e78827-a544-47e4-ac4b-71fb42418fd8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gxwxb" Jul 29 04:13:41.283379 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:41.283336 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4e78827-a544-47e4-ac4b-71fb42418fd8-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gxwxb\" (UID: \"e4e78827-a544-47e4-ac4b-71fb42418fd8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gxwxb" Jul 29 04:13:41.283379 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:41.283379 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4e78827-a544-47e4-ac4b-71fb42418fd8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gxwxb\" (UID: \"e4e78827-a544-47e4-ac4b-71fb42418fd8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gxwxb" Jul 29 04:13:41.283633 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:41.283417 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls26h\" (UniqueName: \"kubernetes.io/projected/e4e78827-a544-47e4-ac4b-71fb42418fd8-kube-api-access-ls26h\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gxwxb\" (UID: \"e4e78827-a544-47e4-ac4b-71fb42418fd8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gxwxb" Jul 29 04:13:41.283633 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:41.283464 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4e78827-a544-47e4-ac4b-71fb42418fd8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gxwxb\" (UID: \"e4e78827-a544-47e4-ac4b-71fb42418fd8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gxwxb" Jul 29 04:13:41.283633 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:41.283487 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e78827-a544-47e4-ac4b-71fb42418fd8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gxwxb\" (UID: \"e4e78827-a544-47e4-ac4b-71fb42418fd8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gxwxb" Jul 29 04:13:41.283633 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:41.283528 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4e78827-a544-47e4-ac4b-71fb42418fd8-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gxwxb\" (UID: \"e4e78827-a544-47e4-ac4b-71fb42418fd8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gxwxb" Jul 29 04:13:41.283835 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:41.283810 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4e78827-a544-47e4-ac4b-71fb42418fd8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gxwxb\" (UID: \"e4e78827-a544-47e4-ac4b-71fb42418fd8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gxwxb" Jul 29 04:13:41.283908 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:41.283895 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4e78827-a544-47e4-ac4b-71fb42418fd8-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gxwxb\" (UID: \"e4e78827-a544-47e4-ac4b-71fb42418fd8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gxwxb" Jul 29 04:13:41.283959 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:41.283921 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4e78827-a544-47e4-ac4b-71fb42418fd8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gxwxb\" (UID: \"e4e78827-a544-47e4-ac4b-71fb42418fd8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gxwxb" Jul 29 04:13:41.285725 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:41.285706 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4e78827-a544-47e4-ac4b-71fb42418fd8-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gxwxb\" (UID: \"e4e78827-a544-47e4-ac4b-71fb42418fd8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gxwxb" Jul 29 04:13:41.285881 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:41.285864 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e78827-a544-47e4-ac4b-71fb42418fd8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gxwxb\" (UID: \"e4e78827-a544-47e4-ac4b-71fb42418fd8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gxwxb" Jul 29 04:13:41.292804 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:41.292780 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls26h\" (UniqueName: \"kubernetes.io/projected/e4e78827-a544-47e4-ac4b-71fb42418fd8-kube-api-access-ls26h\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gxwxb\" (UID: \"e4e78827-a544-47e4-ac4b-71fb42418fd8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gxwxb" Jul 29 04:13:41.438555 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:41.438515 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gxwxb" Jul 29 04:13:41.576421 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:41.576396 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-gxwxb"] Jul 29 04:13:41.578380 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:13:41.578346 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e78827_a544_47e4_ac4b_71fb42418fd8.slice/crio-930315a24521a13025ec06d7b2a37fe042a7e448cc7597f8baabeb57ff289627 WatchSource:0}: Error finding container 930315a24521a13025ec06d7b2a37fe042a7e448cc7597f8baabeb57ff289627: Status 404 returned error can't find the container with id 930315a24521a13025ec06d7b2a37fe042a7e448cc7597f8baabeb57ff289627 Jul 29 04:13:42.236951 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:42.236895 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gxwxb" event={"ID":"e4e78827-a544-47e4-ac4b-71fb42418fd8","Type":"ContainerStarted","Data":"930315a24521a13025ec06d7b2a37fe042a7e448cc7597f8baabeb57ff289627"} Jul 29 04:13:43.242889 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:43.242853 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gxwxb" event={"ID":"e4e78827-a544-47e4-ac4b-71fb42418fd8","Type":"ContainerStarted","Data":"ca38c66209708065cceba1b39fd59f542984ca81371b6ad78a08f1dc7dccc74c"} Jul 29 04:13:43.259722 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:43.259674 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gxwxb" podStartSLOduration=1.2672683679999999 podStartE2EDuration="2.25965675s" totalImagesPullingTime="992.388382ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:13:43.257658047 +0000 UTC m=+604.117032772" watchObservedRunningTime="2026-07-29 04:13:43.25965675 +0000 UTC m=+604.119031474" Jul 29 04:13:51.438670 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:51.438630 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gxwxb" Jul 29 04:13:51.439084 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:51.438696 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gxwxb" Jul 29 04:13:51.451689 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:51.451655 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gxwxb" Jul 29 04:13:52.296628 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:13:52.296598 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gxwxb" Jul 29 04:14:55.570328 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:55.570296 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5fc9448994-dfz7l"] Jul 29 04:14:55.573764 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:55.573740 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fc9448994-dfz7l" Jul 29 04:14:55.579733 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:55.579703 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fc9448994-dfz7l"] Jul 29 04:14:55.648656 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:55.648620 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8e9551f5-88ae-443b-84a9-49e3dbe79ed5-tls-cert\") pod \"authorino-5fc9448994-dfz7l\" (UID: \"8e9551f5-88ae-443b-84a9-49e3dbe79ed5\") " pod="kuadrant-system/authorino-5fc9448994-dfz7l" Jul 29 04:14:55.648824 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:55.648685 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mt8q\" (UniqueName: \"kubernetes.io/projected/8e9551f5-88ae-443b-84a9-49e3dbe79ed5-kube-api-access-6mt8q\") pod \"authorino-5fc9448994-dfz7l\" (UID: \"8e9551f5-88ae-443b-84a9-49e3dbe79ed5\") " pod="kuadrant-system/authorino-5fc9448994-dfz7l" Jul 29 04:14:55.749497 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:55.749455 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8e9551f5-88ae-443b-84a9-49e3dbe79ed5-tls-cert\") pod \"authorino-5fc9448994-dfz7l\" (UID: \"8e9551f5-88ae-443b-84a9-49e3dbe79ed5\") " pod="kuadrant-system/authorino-5fc9448994-dfz7l" Jul 29 04:14:55.749696 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:55.749543 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mt8q\" (UniqueName: \"kubernetes.io/projected/8e9551f5-88ae-443b-84a9-49e3dbe79ed5-kube-api-access-6mt8q\") pod \"authorino-5fc9448994-dfz7l\" (UID: \"8e9551f5-88ae-443b-84a9-49e3dbe79ed5\") " pod="kuadrant-system/authorino-5fc9448994-dfz7l" Jul 29 04:14:55.752065 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:55.752041 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8e9551f5-88ae-443b-84a9-49e3dbe79ed5-tls-cert\") pod \"authorino-5fc9448994-dfz7l\" (UID: \"8e9551f5-88ae-443b-84a9-49e3dbe79ed5\") " pod="kuadrant-system/authorino-5fc9448994-dfz7l" Jul 29 04:14:55.756903 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:55.756883 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mt8q\" (UniqueName: \"kubernetes.io/projected/8e9551f5-88ae-443b-84a9-49e3dbe79ed5-kube-api-access-6mt8q\") pod \"authorino-5fc9448994-dfz7l\" (UID: \"8e9551f5-88ae-443b-84a9-49e3dbe79ed5\") " pod="kuadrant-system/authorino-5fc9448994-dfz7l" Jul 29 04:14:55.884258 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:55.884163 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fc9448994-dfz7l" Jul 29 04:14:56.013073 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:56.013049 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fc9448994-dfz7l"] Jul 29 04:14:56.015400 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:14:56.015372 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e9551f5_88ae_443b_84a9_49e3dbe79ed5.slice/crio-a129eb50b10b54a95ec192e516a79a64d3a979ea9aa575f79fea2ee203fc3753 WatchSource:0}: Error finding container a129eb50b10b54a95ec192e516a79a64d3a979ea9aa575f79fea2ee203fc3753: Status 404 returned error can't find the container with id a129eb50b10b54a95ec192e516a79a64d3a979ea9aa575f79fea2ee203fc3753 Jul 29 04:14:56.016719 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:56.016703 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 04:14:56.585048 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:56.585012 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fc9448994-dfz7l" event={"ID":"8e9551f5-88ae-443b-84a9-49e3dbe79ed5","Type":"ContainerStarted","Data":"f5be49792608c23dc4b116f4b5581e2752662948286d3aed4113dcf4c74f7ae2"} Jul 29 04:14:56.585612 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:56.585585 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fc9448994-dfz7l" event={"ID":"8e9551f5-88ae-443b-84a9-49e3dbe79ed5","Type":"ContainerStarted","Data":"a129eb50b10b54a95ec192e516a79a64d3a979ea9aa575f79fea2ee203fc3753"} Jul 29 04:14:56.599318 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:56.599255 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5fc9448994-dfz7l" podStartSLOduration=1.165203312 podStartE2EDuration="1.599235616s" totalImagesPullingTime="434.032304ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:14:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:14:56.598936308 +0000 UTC m=+677.458311046" watchObservedRunningTime="2026-07-29 04:14:56.599235616 +0000 UTC m=+677.458610340" Jul 29 04:14:56.623515 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:56.623472 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5999dc58c9-cg4bt"] Jul 29 04:14:56.623702 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:56.623637 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5999dc58c9-cg4bt" podUID="6c678bb7-f13e-4f56-a64f-f719943fb1da" containerName="authorino" containerID="cri-o://4d57913af1b814b8882741c13d35c6b33dcad6d3819e1b0438ed163e3965e702" gracePeriod=30 Jul 29 04:14:56.893763 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:56.893591 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5999dc58c9-cg4bt" Jul 29 04:14:56.959220 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:56.959169 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6c678bb7-f13e-4f56-a64f-f719943fb1da-tls-cert\") pod \"6c678bb7-f13e-4f56-a64f-f719943fb1da\" (UID: \"6c678bb7-f13e-4f56-a64f-f719943fb1da\") " Jul 29 04:14:56.959398 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:56.959299 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtfmh\" (UniqueName: \"kubernetes.io/projected/6c678bb7-f13e-4f56-a64f-f719943fb1da-kube-api-access-mtfmh\") pod \"6c678bb7-f13e-4f56-a64f-f719943fb1da\" (UID: \"6c678bb7-f13e-4f56-a64f-f719943fb1da\") " Jul 29 04:14:56.961479 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:56.961423 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c678bb7-f13e-4f56-a64f-f719943fb1da-kube-api-access-mtfmh" (OuterVolumeSpecName: "kube-api-access-mtfmh") pod "6c678bb7-f13e-4f56-a64f-f719943fb1da" (UID: "6c678bb7-f13e-4f56-a64f-f719943fb1da"). InnerVolumeSpecName "kube-api-access-mtfmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:14:56.969686 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:56.969659 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c678bb7-f13e-4f56-a64f-f719943fb1da-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "6c678bb7-f13e-4f56-a64f-f719943fb1da" (UID: "6c678bb7-f13e-4f56-a64f-f719943fb1da"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 04:14:57.060230 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:57.060126 2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6c678bb7-f13e-4f56-a64f-f719943fb1da-tls-cert\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:14:57.060230 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:57.060160 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtfmh\" (UniqueName: \"kubernetes.io/projected/6c678bb7-f13e-4f56-a64f-f719943fb1da-kube-api-access-mtfmh\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:14:57.591343 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:57.591305 2579 generic.go:356] "Generic (PLEG): container finished" podID="6c678bb7-f13e-4f56-a64f-f719943fb1da" containerID="4d57913af1b814b8882741c13d35c6b33dcad6d3819e1b0438ed163e3965e702" exitCode=0 Jul 29 04:14:57.591779 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:57.591403 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5999dc58c9-cg4bt" Jul 29 04:14:57.592583 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:57.592548 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5999dc58c9-cg4bt" event={"ID":"6c678bb7-f13e-4f56-a64f-f719943fb1da","Type":"ContainerDied","Data":"4d57913af1b814b8882741c13d35c6b33dcad6d3819e1b0438ed163e3965e702"} Jul 29 04:14:57.592737 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:57.592679 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5999dc58c9-cg4bt" event={"ID":"6c678bb7-f13e-4f56-a64f-f719943fb1da","Type":"ContainerDied","Data":"2987641c2499c9dfeaa81ffa39abf66ba1b50f92fd8edc06d711aeadf50154e2"} Jul 29 04:14:57.592737 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:57.592718 2579 scope.go:117] "RemoveContainer" containerID="4d57913af1b814b8882741c13d35c6b33dcad6d3819e1b0438ed163e3965e702" Jul 29 04:14:57.602690 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:57.602672 2579 scope.go:117] "RemoveContainer" containerID="4d57913af1b814b8882741c13d35c6b33dcad6d3819e1b0438ed163e3965e702" Jul 29 04:14:57.603056 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:14:57.603021 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d57913af1b814b8882741c13d35c6b33dcad6d3819e1b0438ed163e3965e702\": container with ID starting with 4d57913af1b814b8882741c13d35c6b33dcad6d3819e1b0438ed163e3965e702 not found: ID does not exist" containerID="4d57913af1b814b8882741c13d35c6b33dcad6d3819e1b0438ed163e3965e702" Jul 29 04:14:57.603150 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:57.603051 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d57913af1b814b8882741c13d35c6b33dcad6d3819e1b0438ed163e3965e702"} err="failed to get container status \"4d57913af1b814b8882741c13d35c6b33dcad6d3819e1b0438ed163e3965e702\": rpc error: code = NotFound desc = could not find container \"4d57913af1b814b8882741c13d35c6b33dcad6d3819e1b0438ed163e3965e702\": container with ID starting with 4d57913af1b814b8882741c13d35c6b33dcad6d3819e1b0438ed163e3965e702 not found: ID does not exist" Jul 29 04:14:57.615505 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:57.615479 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5999dc58c9-cg4bt"] Jul 29 04:14:57.620737 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:57.620715 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5999dc58c9-cg4bt"] Jul 29 04:14:59.592262 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:14:59.592230 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c678bb7-f13e-4f56-a64f-f719943fb1da" path="/var/lib/kubelet/pods/6c678bb7-f13e-4f56-a64f-f719943fb1da/volumes" Jul 29 04:15:00.148162 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:00.148124 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754975-d9tln"] Jul 29 04:15:00.148530 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:00.148516 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c678bb7-f13e-4f56-a64f-f719943fb1da" containerName="authorino" Jul 29 04:15:00.148589 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:00.148533 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c678bb7-f13e-4f56-a64f-f719943fb1da" containerName="authorino" Jul 29 04:15:00.148630 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:00.148594 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c678bb7-f13e-4f56-a64f-f719943fb1da" containerName="authorino" Jul 29 04:15:00.152901 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:00.152886 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754975-d9tln" Jul 29 04:15:00.155240 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:00.155202 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mxv5w\"" Jul 29 04:15:00.158638 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:00.158615 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754975-d9tln"] Jul 29 04:15:00.165567 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:00.165517 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754975-d9tln" Jul 29 04:15:00.297528 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:00.297499 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754975-d9tln"] Jul 29 04:15:00.299311 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:15:00.299280 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5d0d12_d04f_4cd0_bf20_bf7cd3b7b688.slice/crio-1273b330a7b20980ed6219585ac8fdb3e27ee44887aeac1241cd49dac93f86b8 WatchSource:0}: Error finding container 1273b330a7b20980ed6219585ac8fdb3e27ee44887aeac1241cd49dac93f86b8: Status 404 returned error can't find the container with id 1273b330a7b20980ed6219585ac8fdb3e27ee44887aeac1241cd49dac93f86b8 Jul 29 04:15:00.606780 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:00.606745 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754975-d9tln" event={"ID":"7d5d0d12-d04f-4cd0-bf20-bf7cd3b7b688","Type":"ContainerStarted","Data":"1273b330a7b20980ed6219585ac8fdb3e27ee44887aeac1241cd49dac93f86b8"} Jul 29 04:15:03.621446 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:03.621362 2579 generic.go:356] "Generic (PLEG): container finished" podID="7d5d0d12-d04f-4cd0-bf20-bf7cd3b7b688" containerID="54595acf85e21fcf621493040ea2782f444b7360a93fc840d4f8d8a3f3b9b4e0" exitCode=0 Jul 29 04:15:03.621822 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:03.621445 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754975-d9tln" event={"ID":"7d5d0d12-d04f-4cd0-bf20-bf7cd3b7b688","Type":"ContainerDied","Data":"54595acf85e21fcf621493040ea2782f444b7360a93fc840d4f8d8a3f3b9b4e0"} Jul 29 04:15:04.770591 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:04.770551 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754975-d9tln" Jul 29 04:15:05.632897 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:05.632862 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754975-d9tln" event={"ID":"7d5d0d12-d04f-4cd0-bf20-bf7cd3b7b688","Type":"ContainerDied","Data":"1273b330a7b20980ed6219585ac8fdb3e27ee44887aeac1241cd49dac93f86b8"} Jul 29 04:15:05.632897 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:05.632888 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754975-d9tln" Jul 29 04:15:05.633127 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:05.632909 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1273b330a7b20980ed6219585ac8fdb3e27ee44887aeac1241cd49dac93f86b8" Jul 29 04:15:45.419393 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:45.419325 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b9ff676b5-ljc2v"] Jul 29 04:15:45.419921 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:45.419623 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7b9ff676b5-ljc2v" podUID="235aea81-0258-4cb1-99a7-0258e6ed02df" containerName="manager" containerID="cri-o://e3fe0c97600c4616401b31c881725bcc16702b56e7c14199a75f0aa9010d70f1" gracePeriod=30 Jul 29 04:15:45.672493 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:45.672434 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b9ff676b5-ljc2v" Jul 29 04:15:45.805426 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:45.805391 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8t6d\" (UniqueName: \"kubernetes.io/projected/235aea81-0258-4cb1-99a7-0258e6ed02df-kube-api-access-c8t6d\") pod \"235aea81-0258-4cb1-99a7-0258e6ed02df\" (UID: \"235aea81-0258-4cb1-99a7-0258e6ed02df\") " Jul 29 04:15:45.805590 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:45.805498 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/235aea81-0258-4cb1-99a7-0258e6ed02df-webhook-cert\") pod \"235aea81-0258-4cb1-99a7-0258e6ed02df\" (UID: \"235aea81-0258-4cb1-99a7-0258e6ed02df\") " Jul 29 04:15:45.807563 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:45.807535 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/235aea81-0258-4cb1-99a7-0258e6ed02df-kube-api-access-c8t6d" (OuterVolumeSpecName: "kube-api-access-c8t6d") pod "235aea81-0258-4cb1-99a7-0258e6ed02df" (UID: "235aea81-0258-4cb1-99a7-0258e6ed02df"). InnerVolumeSpecName "kube-api-access-c8t6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:15:45.807683 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:45.807584 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/235aea81-0258-4cb1-99a7-0258e6ed02df-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "235aea81-0258-4cb1-99a7-0258e6ed02df" (UID: "235aea81-0258-4cb1-99a7-0258e6ed02df"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 04:15:45.825682 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:45.825647 2579 generic.go:356] "Generic (PLEG): container finished" podID="235aea81-0258-4cb1-99a7-0258e6ed02df" containerID="e3fe0c97600c4616401b31c881725bcc16702b56e7c14199a75f0aa9010d70f1" exitCode=0 Jul 29 04:15:45.825682 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:45.825706 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b9ff676b5-ljc2v" Jul 29 04:15:45.825956 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:45.825737 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b9ff676b5-ljc2v" event={"ID":"235aea81-0258-4cb1-99a7-0258e6ed02df","Type":"ContainerDied","Data":"e3fe0c97600c4616401b31c881725bcc16702b56e7c14199a75f0aa9010d70f1"} Jul 29 04:15:45.825956 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:45.825827 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b9ff676b5-ljc2v" event={"ID":"235aea81-0258-4cb1-99a7-0258e6ed02df","Type":"ContainerDied","Data":"f4e6ff416b22129aeed0eae2ee141bb8b9778f8bfad1026b5231e92dcdd4521c"} Jul 29 04:15:45.825956 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:45.825858 2579 scope.go:117] "RemoveContainer" containerID="e3fe0c97600c4616401b31c881725bcc16702b56e7c14199a75f0aa9010d70f1" Jul 29 04:15:45.836102 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:45.836087 2579 scope.go:117] "RemoveContainer" containerID="e3fe0c97600c4616401b31c881725bcc16702b56e7c14199a75f0aa9010d70f1" Jul 29 04:15:45.836425 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:15:45.836400 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3fe0c97600c4616401b31c881725bcc16702b56e7c14199a75f0aa9010d70f1\": container with ID starting with e3fe0c97600c4616401b31c881725bcc16702b56e7c14199a75f0aa9010d70f1 not found: ID does not exist" containerID="e3fe0c97600c4616401b31c881725bcc16702b56e7c14199a75f0aa9010d70f1" Jul 29 04:15:45.836605 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:45.836431 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3fe0c97600c4616401b31c881725bcc16702b56e7c14199a75f0aa9010d70f1"} err="failed to get container status \"e3fe0c97600c4616401b31c881725bcc16702b56e7c14199a75f0aa9010d70f1\": rpc error: code = NotFound desc = could not find container \"e3fe0c97600c4616401b31c881725bcc16702b56e7c14199a75f0aa9010d70f1\": container with ID starting with e3fe0c97600c4616401b31c881725bcc16702b56e7c14199a75f0aa9010d70f1 not found: ID does not exist" Jul 29 04:15:45.846923 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:45.846903 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b9ff676b5-ljc2v"] Jul 29 04:15:45.850839 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:45.850820 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7b9ff676b5-ljc2v"] Jul 29 04:15:45.906657 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:45.906624 2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/235aea81-0258-4cb1-99a7-0258e6ed02df-webhook-cert\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:15:45.906657 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:45.906653 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8t6d\" (UniqueName: \"kubernetes.io/projected/235aea81-0258-4cb1-99a7-0258e6ed02df-kube-api-access-c8t6d\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:15:47.591012 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:15:47.590980 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="235aea81-0258-4cb1-99a7-0258e6ed02df" path="/var/lib/kubelet/pods/235aea81-0258-4cb1-99a7-0258e6ed02df/volumes" Jul 29 04:16:46.664229 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:16:46.664175 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b9ff676b5-cffqz"] Jul 29 04:16:46.664638 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:16:46.664537 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="235aea81-0258-4cb1-99a7-0258e6ed02df" containerName="manager" Jul 29 04:16:46.664638 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:16:46.664548 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="235aea81-0258-4cb1-99a7-0258e6ed02df" containerName="manager" Jul 29 04:16:46.664638 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:16:46.664560 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d5d0d12-d04f-4cd0-bf20-bf7cd3b7b688" containerName="cleanup" Jul 29 04:16:46.664638 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:16:46.664566 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d5d0d12-d04f-4cd0-bf20-bf7cd3b7b688" containerName="cleanup" Jul 29 04:16:46.664765 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:16:46.664643 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="235aea81-0258-4cb1-99a7-0258e6ed02df" containerName="manager" Jul 29 04:16:46.664765 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:16:46.664654 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d5d0d12-d04f-4cd0-bf20-bf7cd3b7b688" containerName="cleanup" Jul 29 04:16:46.668479 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:16:46.668461 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b9ff676b5-cffqz" Jul 29 04:16:46.672001 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:16:46.671980 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-45vj7\"" Jul 29 04:16:46.673186 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:16:46.673169 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 29 04:16:46.673440 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:16:46.673421 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 29 04:16:46.689570 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:16:46.689543 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b9ff676b5-cffqz"] Jul 29 04:16:46.743577 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:16:46.743546 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/43664fa9-44a9-4a42-bb37-18377211ee7f-webhook-cert\") pod \"maas-controller-7b9ff676b5-cffqz\" (UID: \"43664fa9-44a9-4a42-bb37-18377211ee7f\") " pod="opendatahub/maas-controller-7b9ff676b5-cffqz" Jul 29 04:16:46.743778 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:16:46.743581 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd6kv\" (UniqueName: \"kubernetes.io/projected/43664fa9-44a9-4a42-bb37-18377211ee7f-kube-api-access-qd6kv\") pod \"maas-controller-7b9ff676b5-cffqz\" (UID: \"43664fa9-44a9-4a42-bb37-18377211ee7f\") " pod="opendatahub/maas-controller-7b9ff676b5-cffqz" Jul 29 04:16:46.844538 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:16:46.844503 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/43664fa9-44a9-4a42-bb37-18377211ee7f-webhook-cert\") pod \"maas-controller-7b9ff676b5-cffqz\" (UID: \"43664fa9-44a9-4a42-bb37-18377211ee7f\") " pod="opendatahub/maas-controller-7b9ff676b5-cffqz" Jul 29 04:16:46.844538 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:16:46.844543 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd6kv\" (UniqueName: \"kubernetes.io/projected/43664fa9-44a9-4a42-bb37-18377211ee7f-kube-api-access-qd6kv\") pod \"maas-controller-7b9ff676b5-cffqz\" (UID: \"43664fa9-44a9-4a42-bb37-18377211ee7f\") " pod="opendatahub/maas-controller-7b9ff676b5-cffqz" Jul 29 04:16:46.846947 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:16:46.846917 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/43664fa9-44a9-4a42-bb37-18377211ee7f-webhook-cert\") pod \"maas-controller-7b9ff676b5-cffqz\" (UID: \"43664fa9-44a9-4a42-bb37-18377211ee7f\") " pod="opendatahub/maas-controller-7b9ff676b5-cffqz" Jul 29 04:16:46.852598 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:16:46.852572 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd6kv\" (UniqueName: \"kubernetes.io/projected/43664fa9-44a9-4a42-bb37-18377211ee7f-kube-api-access-qd6kv\") pod \"maas-controller-7b9ff676b5-cffqz\" (UID: \"43664fa9-44a9-4a42-bb37-18377211ee7f\") " pod="opendatahub/maas-controller-7b9ff676b5-cffqz" Jul 29 04:16:46.978623 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:16:46.978581 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b9ff676b5-cffqz" Jul 29 04:16:47.110288 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:16:47.110264 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b9ff676b5-cffqz"] Jul 29 04:16:47.112336 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:16:47.112300 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43664fa9_44a9_4a42_bb37_18377211ee7f.slice/crio-6b61d95249d8a66d051311c9975677b33a2dc47465be4be6253389a95a9d7d0c WatchSource:0}: Error finding container 6b61d95249d8a66d051311c9975677b33a2dc47465be4be6253389a95a9d7d0c: Status 404 returned error can't find the container with id 6b61d95249d8a66d051311c9975677b33a2dc47465be4be6253389a95a9d7d0c Jul 29 04:16:48.111215 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:16:48.111168 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b9ff676b5-cffqz" event={"ID":"43664fa9-44a9-4a42-bb37-18377211ee7f","Type":"ContainerStarted","Data":"1f000045ef3c129255a9ec69c4223bf47e34dbe9c25c331e679edf9ca0fd2261"} Jul 29 04:16:48.111620 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:16:48.111239 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b9ff676b5-cffqz" event={"ID":"43664fa9-44a9-4a42-bb37-18377211ee7f","Type":"ContainerStarted","Data":"6b61d95249d8a66d051311c9975677b33a2dc47465be4be6253389a95a9d7d0c"} Jul 29 04:16:48.111620 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:16:48.111264 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b9ff676b5-cffqz" Jul 29 04:16:48.126497 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:16:48.126448 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7b9ff676b5-cffqz" podStartSLOduration=1.675230774 podStartE2EDuration="2.126431542s" totalImagesPullingTime="451.200768ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:16:48.125259979 +0000 UTC m=+788.984634703" watchObservedRunningTime="2026-07-29 04:16:48.126431542 +0000 UTC m=+788.985806266" Jul 29 04:16:59.118826 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:16:59.118796 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b9ff676b5-cffqz" Jul 29 04:17:03.732890 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:03.732800 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b9ff676b5-cffqz"] Jul 29 04:17:03.733309 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:03.733018 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7b9ff676b5-cffqz" podUID="43664fa9-44a9-4a42-bb37-18377211ee7f" containerName="manager" containerID="cri-o://1f000045ef3c129255a9ec69c4223bf47e34dbe9c25c331e679edf9ca0fd2261" gracePeriod=30 Jul 29 04:17:03.982242 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:03.982198 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b9ff676b5-cffqz" Jul 29 04:17:03.999192 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:03.999074 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/43664fa9-44a9-4a42-bb37-18377211ee7f-webhook-cert\") pod \"43664fa9-44a9-4a42-bb37-18377211ee7f\" (UID: \"43664fa9-44a9-4a42-bb37-18377211ee7f\") " Jul 29 04:17:03.999192 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:03.999143 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qd6kv\" (UniqueName: \"kubernetes.io/projected/43664fa9-44a9-4a42-bb37-18377211ee7f-kube-api-access-qd6kv\") pod \"43664fa9-44a9-4a42-bb37-18377211ee7f\" (UID: \"43664fa9-44a9-4a42-bb37-18377211ee7f\") " Jul 29 04:17:04.001991 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:04.001570 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43664fa9-44a9-4a42-bb37-18377211ee7f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "43664fa9-44a9-4a42-bb37-18377211ee7f" (UID: "43664fa9-44a9-4a42-bb37-18377211ee7f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 04:17:04.001991 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:04.001909 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43664fa9-44a9-4a42-bb37-18377211ee7f-kube-api-access-qd6kv" (OuterVolumeSpecName: "kube-api-access-qd6kv") pod "43664fa9-44a9-4a42-bb37-18377211ee7f" (UID: "43664fa9-44a9-4a42-bb37-18377211ee7f"). InnerVolumeSpecName "kube-api-access-qd6kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:17:04.099794 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:04.099747 2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/43664fa9-44a9-4a42-bb37-18377211ee7f-webhook-cert\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:17:04.099794 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:04.099781 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd6kv\" (UniqueName: \"kubernetes.io/projected/43664fa9-44a9-4a42-bb37-18377211ee7f-kube-api-access-qd6kv\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:17:04.185797 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:04.185760 2579 generic.go:356] "Generic (PLEG): container finished" podID="43664fa9-44a9-4a42-bb37-18377211ee7f" containerID="1f000045ef3c129255a9ec69c4223bf47e34dbe9c25c331e679edf9ca0fd2261" exitCode=0 Jul 29 04:17:04.185964 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:04.185823 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b9ff676b5-cffqz" Jul 29 04:17:04.185964 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:04.185852 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b9ff676b5-cffqz" event={"ID":"43664fa9-44a9-4a42-bb37-18377211ee7f","Type":"ContainerDied","Data":"1f000045ef3c129255a9ec69c4223bf47e34dbe9c25c331e679edf9ca0fd2261"} Jul 29 04:17:04.186051 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:04.185965 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b9ff676b5-cffqz" event={"ID":"43664fa9-44a9-4a42-bb37-18377211ee7f","Type":"ContainerDied","Data":"6b61d95249d8a66d051311c9975677b33a2dc47465be4be6253389a95a9d7d0c"} Jul 29 04:17:04.186051 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:04.186002 2579 scope.go:117] "RemoveContainer" containerID="1f000045ef3c129255a9ec69c4223bf47e34dbe9c25c331e679edf9ca0fd2261" Jul 29 04:17:04.206578 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:04.206557 2579 scope.go:117] "RemoveContainer" containerID="1f000045ef3c129255a9ec69c4223bf47e34dbe9c25c331e679edf9ca0fd2261" Jul 29 04:17:04.206859 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:17:04.206836 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f000045ef3c129255a9ec69c4223bf47e34dbe9c25c331e679edf9ca0fd2261\": container with ID starting with 1f000045ef3c129255a9ec69c4223bf47e34dbe9c25c331e679edf9ca0fd2261 not found: ID does not exist" containerID="1f000045ef3c129255a9ec69c4223bf47e34dbe9c25c331e679edf9ca0fd2261" Jul 29 04:17:04.206928 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:04.206866 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f000045ef3c129255a9ec69c4223bf47e34dbe9c25c331e679edf9ca0fd2261"} err="failed to get container status \"1f000045ef3c129255a9ec69c4223bf47e34dbe9c25c331e679edf9ca0fd2261\": rpc error: code = NotFound desc = could not find container \"1f000045ef3c129255a9ec69c4223bf47e34dbe9c25c331e679edf9ca0fd2261\": container with ID starting with 1f000045ef3c129255a9ec69c4223bf47e34dbe9c25c331e679edf9ca0fd2261 not found: ID does not exist" Jul 29 04:17:04.212765 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:04.212740 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b9ff676b5-cffqz"] Jul 29 04:17:04.216867 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:04.216842 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7b9ff676b5-cffqz"] Jul 29 04:17:04.965800 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:04.965772 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b9ff676b5-dgmvr"] Jul 29 04:17:04.966199 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:04.966140 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43664fa9-44a9-4a42-bb37-18377211ee7f" containerName="manager" Jul 29 04:17:04.966199 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:04.966159 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="43664fa9-44a9-4a42-bb37-18377211ee7f" containerName="manager" Jul 29 04:17:04.966292 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:04.966257 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="43664fa9-44a9-4a42-bb37-18377211ee7f" containerName="manager" Jul 29 04:17:04.969240 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:04.969198 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b9ff676b5-dgmvr" Jul 29 04:17:04.971601 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:04.971583 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 29 04:17:04.972466 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:04.972448 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-45vj7\"" Jul 29 04:17:04.972548 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:04.972504 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 29 04:17:04.978174 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:04.978150 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b9ff676b5-dgmvr"] Jul 29 04:17:05.010260 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:05.010220 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/793f13a7-a5e3-4f3a-a0d1-3d5caba57f23-webhook-cert\") pod \"maas-controller-7b9ff676b5-dgmvr\" (UID: \"793f13a7-a5e3-4f3a-a0d1-3d5caba57f23\") " pod="opendatahub/maas-controller-7b9ff676b5-dgmvr" Jul 29 04:17:05.010427 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:05.010269 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79slp\" (UniqueName: \"kubernetes.io/projected/793f13a7-a5e3-4f3a-a0d1-3d5caba57f23-kube-api-access-79slp\") pod \"maas-controller-7b9ff676b5-dgmvr\" (UID: \"793f13a7-a5e3-4f3a-a0d1-3d5caba57f23\") " pod="opendatahub/maas-controller-7b9ff676b5-dgmvr" Jul 29 04:17:05.111333 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:05.111298 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/793f13a7-a5e3-4f3a-a0d1-3d5caba57f23-webhook-cert\") pod \"maas-controller-7b9ff676b5-dgmvr\" (UID: \"793f13a7-a5e3-4f3a-a0d1-3d5caba57f23\") " pod="opendatahub/maas-controller-7b9ff676b5-dgmvr" Jul 29 04:17:05.111333 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:05.111338 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79slp\" (UniqueName: \"kubernetes.io/projected/793f13a7-a5e3-4f3a-a0d1-3d5caba57f23-kube-api-access-79slp\") pod \"maas-controller-7b9ff676b5-dgmvr\" (UID: \"793f13a7-a5e3-4f3a-a0d1-3d5caba57f23\") " pod="opendatahub/maas-controller-7b9ff676b5-dgmvr" Jul 29 04:17:05.113770 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:05.113747 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/793f13a7-a5e3-4f3a-a0d1-3d5caba57f23-webhook-cert\") pod \"maas-controller-7b9ff676b5-dgmvr\" (UID: \"793f13a7-a5e3-4f3a-a0d1-3d5caba57f23\") " pod="opendatahub/maas-controller-7b9ff676b5-dgmvr" Jul 29 04:17:05.119707 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:05.119678 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79slp\" (UniqueName: \"kubernetes.io/projected/793f13a7-a5e3-4f3a-a0d1-3d5caba57f23-kube-api-access-79slp\") pod \"maas-controller-7b9ff676b5-dgmvr\" (UID: \"793f13a7-a5e3-4f3a-a0d1-3d5caba57f23\") " pod="opendatahub/maas-controller-7b9ff676b5-dgmvr" Jul 29 04:17:05.279766 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:05.279675 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b9ff676b5-dgmvr" Jul 29 04:17:05.413063 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:05.413035 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b9ff676b5-dgmvr"] Jul 29 04:17:05.415006 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:17:05.414977 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod793f13a7_a5e3_4f3a_a0d1_3d5caba57f23.slice/crio-660b13e884d0642e1758009f871cd2f8054cb3a8ce70031fa83df6c77c582905 WatchSource:0}: Error finding container 660b13e884d0642e1758009f871cd2f8054cb3a8ce70031fa83df6c77c582905: Status 404 returned error can't find the container with id 660b13e884d0642e1758009f871cd2f8054cb3a8ce70031fa83df6c77c582905 Jul 29 04:17:05.591768 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:05.591693 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43664fa9-44a9-4a42-bb37-18377211ee7f" path="/var/lib/kubelet/pods/43664fa9-44a9-4a42-bb37-18377211ee7f/volumes" Jul 29 04:17:06.198557 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:06.198520 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b9ff676b5-dgmvr" event={"ID":"793f13a7-a5e3-4f3a-a0d1-3d5caba57f23","Type":"ContainerStarted","Data":"db73bf7fb4eefe8d9ad41f9e95c1795f72e27e802ae2e3722b972427959d2ca3"} Jul 29 04:17:06.198933 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:06.198581 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b9ff676b5-dgmvr" event={"ID":"793f13a7-a5e3-4f3a-a0d1-3d5caba57f23","Type":"ContainerStarted","Data":"660b13e884d0642e1758009f871cd2f8054cb3a8ce70031fa83df6c77c582905"} Jul 29 04:17:06.198933 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:06.198630 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b9ff676b5-dgmvr" Jul 29 04:17:06.214276 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:06.214196 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7b9ff676b5-dgmvr" podStartSLOduration=1.80556823 podStartE2EDuration="2.214178247s" totalImagesPullingTime="408.610017ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:17:06.213937831 +0000 UTC m=+807.073312590" watchObservedRunningTime="2026-07-29 04:17:06.214178247 +0000 UTC m=+807.073552971" Jul 29 04:17:17.206193 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:17:17.206162 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b9ff676b5-dgmvr" Jul 29 04:25:28.063007 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:25:28.062967 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szn9p"] Jul 29 04:25:28.066344 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:25:28.066327 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szn9p" Jul 29 04:25:28.068840 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:25:28.068819 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-nkp54\"" Jul 29 04:25:28.087112 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:25:28.087083 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szn9p"] Jul 29 04:25:28.142728 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:25:28.142694 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2smh9\" (UniqueName: \"kubernetes.io/projected/9a2a3d4c-6605-4e87-bb39-ec1402d0d894-kube-api-access-2smh9\") pod \"kuadrant-operator-controller-manager-86878c769f-szn9p\" (UID: \"9a2a3d4c-6605-4e87-bb39-ec1402d0d894\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szn9p" Jul 29 04:25:28.142935 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:25:28.142812 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9a2a3d4c-6605-4e87-bb39-ec1402d0d894-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-szn9p\" (UID: \"9a2a3d4c-6605-4e87-bb39-ec1402d0d894\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szn9p" Jul 29 04:25:28.243799 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:25:28.243757 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9a2a3d4c-6605-4e87-bb39-ec1402d0d894-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-szn9p\" (UID: \"9a2a3d4c-6605-4e87-bb39-ec1402d0d894\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szn9p" Jul 29 04:25:28.243960 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:25:28.243835 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2smh9\" (UniqueName: \"kubernetes.io/projected/9a2a3d4c-6605-4e87-bb39-ec1402d0d894-kube-api-access-2smh9\") pod \"kuadrant-operator-controller-manager-86878c769f-szn9p\" (UID: \"9a2a3d4c-6605-4e87-bb39-ec1402d0d894\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szn9p" Jul 29 04:25:28.244261 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:25:28.244240 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9a2a3d4c-6605-4e87-bb39-ec1402d0d894-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-szn9p\" (UID: \"9a2a3d4c-6605-4e87-bb39-ec1402d0d894\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szn9p" Jul 29 04:25:28.253437 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:25:28.253412 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2smh9\" (UniqueName: \"kubernetes.io/projected/9a2a3d4c-6605-4e87-bb39-ec1402d0d894-kube-api-access-2smh9\") pod \"kuadrant-operator-controller-manager-86878c769f-szn9p\" (UID: \"9a2a3d4c-6605-4e87-bb39-ec1402d0d894\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szn9p" Jul 29 04:25:28.376716 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:25:28.376622 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szn9p" Jul 29 04:25:28.519833 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:25:28.519799 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szn9p"] Jul 29 04:25:28.522255 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:25:28.522228 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a2a3d4c_6605_4e87_bb39_ec1402d0d894.slice/crio-1b46ff09267b09080650e49ec01d6847d5f5200676bf485d56dc90c78f0d7545 WatchSource:0}: Error finding container 1b46ff09267b09080650e49ec01d6847d5f5200676bf485d56dc90c78f0d7545: Status 404 returned error can't find the container with id 1b46ff09267b09080650e49ec01d6847d5f5200676bf485d56dc90c78f0d7545 Jul 29 04:25:28.539876 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:25:28.539851 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szn9p" event={"ID":"9a2a3d4c-6605-4e87-bb39-ec1402d0d894","Type":"ContainerStarted","Data":"1b46ff09267b09080650e49ec01d6847d5f5200676bf485d56dc90c78f0d7545"} Jul 29 04:25:29.546229 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:25:29.546178 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szn9p" event={"ID":"9a2a3d4c-6605-4e87-bb39-ec1402d0d894","Type":"ContainerStarted","Data":"fdf5d264d9dc4bf26a6e17d41092f7694ffc80efa1fb44bba7f4b42f476f66aa"} Jul 29 04:25:29.546708 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:25:29.546273 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szn9p" Jul 29 04:25:29.582402 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:25:29.582343 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szn9p" podStartSLOduration=1.5823266980000001 podStartE2EDuration="1.582326698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:25:29.579525302 +0000 UTC m=+1310.438900025" watchObservedRunningTime="2026-07-29 04:25:29.582326698 +0000 UTC m=+1310.441701421" Jul 29 04:25:40.552483 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:25:40.552447 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szn9p" Jul 29 04:28:08.834456 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:08.834417 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9"] Jul 29 04:28:08.838168 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:08.838151 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" Jul 29 04:28:08.840346 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:08.840323 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-c3af2471-kserve-self-signed-certs\"" Jul 29 04:28:08.862974 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:08.862943 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9"] Jul 29 04:28:08.869118 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:08.869092 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/488c2811-20b2-4919-b601-d9c63d42a5a8-dshm\") pod \"e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9\" (UID: \"488c2811-20b2-4919-b601-d9c63d42a5a8\") " pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" Jul 29 04:28:08.869318 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:08.869139 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/488c2811-20b2-4919-b601-d9c63d42a5a8-home\") pod \"e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9\" (UID: \"488c2811-20b2-4919-b601-d9c63d42a5a8\") " pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" Jul 29 04:28:08.869318 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:08.869183 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/488c2811-20b2-4919-b601-d9c63d42a5a8-model-cache\") pod \"e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9\" (UID: \"488c2811-20b2-4919-b601-d9c63d42a5a8\") " pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" Jul 29 04:28:08.869318 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:08.869259 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/488c2811-20b2-4919-b601-d9c63d42a5a8-tmp-dir\") pod \"e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9\" (UID: \"488c2811-20b2-4919-b601-d9c63d42a5a8\") " pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" Jul 29 04:28:08.869466 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:08.869323 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/488c2811-20b2-4919-b601-d9c63d42a5a8-tls-certs\") pod \"e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9\" (UID: \"488c2811-20b2-4919-b601-d9c63d42a5a8\") " pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" Jul 29 04:28:08.869466 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:08.869358 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn4s6\" (UniqueName: \"kubernetes.io/projected/488c2811-20b2-4919-b601-d9c63d42a5a8-kube-api-access-qn4s6\") pod \"e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9\" (UID: \"488c2811-20b2-4919-b601-d9c63d42a5a8\") " pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" Jul 29 04:28:08.970859 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:08.970815 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/488c2811-20b2-4919-b601-d9c63d42a5a8-tls-certs\") pod \"e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9\" (UID: \"488c2811-20b2-4919-b601-d9c63d42a5a8\") " pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" Jul 29 04:28:08.971036 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:08.970874 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn4s6\" (UniqueName: \"kubernetes.io/projected/488c2811-20b2-4919-b601-d9c63d42a5a8-kube-api-access-qn4s6\") pod \"e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9\" (UID: \"488c2811-20b2-4919-b601-d9c63d42a5a8\") " pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" Jul 29 04:28:08.971036 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:08.970912 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/488c2811-20b2-4919-b601-d9c63d42a5a8-dshm\") pod \"e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9\" (UID: \"488c2811-20b2-4919-b601-d9c63d42a5a8\") " pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" Jul 29 04:28:08.971036 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:08.970940 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/488c2811-20b2-4919-b601-d9c63d42a5a8-home\") pod \"e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9\" (UID: \"488c2811-20b2-4919-b601-d9c63d42a5a8\") " pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" Jul 29 04:28:08.971036 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:08.970957 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/488c2811-20b2-4919-b601-d9c63d42a5a8-model-cache\") pod \"e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9\" (UID: \"488c2811-20b2-4919-b601-d9c63d42a5a8\") " pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" Jul 29 04:28:08.971036 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:08.970973 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/488c2811-20b2-4919-b601-d9c63d42a5a8-tmp-dir\") pod \"e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9\" (UID: \"488c2811-20b2-4919-b601-d9c63d42a5a8\") " pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" Jul 29 04:28:08.971449 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:08.971420 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/488c2811-20b2-4919-b601-d9c63d42a5a8-home\") pod \"e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9\" (UID: \"488c2811-20b2-4919-b601-d9c63d42a5a8\") " pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" Jul 29 04:28:08.971559 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:08.971514 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/488c2811-20b2-4919-b601-d9c63d42a5a8-tmp-dir\") pod \"e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9\" (UID: \"488c2811-20b2-4919-b601-d9c63d42a5a8\") " pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" Jul 29 04:28:08.971559 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:08.971544 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/488c2811-20b2-4919-b601-d9c63d42a5a8-model-cache\") pod \"e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9\" (UID: \"488c2811-20b2-4919-b601-d9c63d42a5a8\") " pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" Jul 29 04:28:08.973172 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:08.973142 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/488c2811-20b2-4919-b601-d9c63d42a5a8-dshm\") pod \"e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9\" (UID: \"488c2811-20b2-4919-b601-d9c63d42a5a8\") " pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" Jul 29 04:28:08.973485 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:08.973461 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/488c2811-20b2-4919-b601-d9c63d42a5a8-tls-certs\") pod \"e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9\" (UID: \"488c2811-20b2-4919-b601-d9c63d42a5a8\") " pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" Jul 29 04:28:08.978745 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:08.978725 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn4s6\" (UniqueName: \"kubernetes.io/projected/488c2811-20b2-4919-b601-d9c63d42a5a8-kube-api-access-qn4s6\") pod \"e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9\" (UID: \"488c2811-20b2-4919-b601-d9c63d42a5a8\") " pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" Jul 29 04:28:09.148046 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:09.147960 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" Jul 29 04:28:09.285787 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:09.285762 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9"] Jul 29 04:28:09.287740 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:28:09.287710 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod488c2811_20b2_4919_b601_d9c63d42a5a8.slice/crio-e34ab0b48060832e8fa7e5cdeaf0960386ffd868ea12b027311eff1ad68d17e5 WatchSource:0}: Error finding container e34ab0b48060832e8fa7e5cdeaf0960386ffd868ea12b027311eff1ad68d17e5: Status 404 returned error can't find the container with id e34ab0b48060832e8fa7e5cdeaf0960386ffd868ea12b027311eff1ad68d17e5 Jul 29 04:28:09.315021 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:09.314991 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" event={"ID":"488c2811-20b2-4919-b601-d9c63d42a5a8","Type":"ContainerStarted","Data":"e34ab0b48060832e8fa7e5cdeaf0960386ffd868ea12b027311eff1ad68d17e5"} Jul 29 04:28:10.320701 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:10.320667 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" event={"ID":"488c2811-20b2-4919-b601-d9c63d42a5a8","Type":"ContainerStarted","Data":"dff877afe18018003a4beb8fb478dbaf79340d1002fc3b0be60971af18f66d21"} Jul 29 04:28:10.337244 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:10.337174 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" podStartSLOduration=2.337158277 podStartE2EDuration="2.337158277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:28:10.335609808 +0000 UTC m=+1471.194984527" watchObservedRunningTime="2026-07-29 04:28:10.337158277 +0000 UTC m=+1471.196533000" Jul 29 04:28:19.148417 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:19.148330 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" Jul 29 04:28:19.148417 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:19.148383 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" Jul 29 04:28:19.160795 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:19.160768 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" Jul 29 04:28:19.375243 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:19.375197 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" Jul 29 04:28:44.381434 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:44.381396 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9"] Jul 29 04:28:59.401375 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:59.401320 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" podUID="488c2811-20b2-4919-b601-d9c63d42a5a8" containerName="main" containerID="cri-o://dff877afe18018003a4beb8fb478dbaf79340d1002fc3b0be60971af18f66d21" gracePeriod=45 Jul 29 04:28:59.568236 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:59.568184 2579 generic.go:356] "Generic (PLEG): container finished" podID="488c2811-20b2-4919-b601-d9c63d42a5a8" containerID="dff877afe18018003a4beb8fb478dbaf79340d1002fc3b0be60971af18f66d21" exitCode=0 Jul 29 04:28:59.568398 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:59.568247 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" event={"ID":"488c2811-20b2-4919-b601-d9c63d42a5a8","Type":"ContainerDied","Data":"dff877afe18018003a4beb8fb478dbaf79340d1002fc3b0be60971af18f66d21"} Jul 29 04:28:59.668094 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:59.668018 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" Jul 29 04:28:59.733509 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:59.733476 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/488c2811-20b2-4919-b601-d9c63d42a5a8-tls-certs\") pod \"488c2811-20b2-4919-b601-d9c63d42a5a8\" (UID: \"488c2811-20b2-4919-b601-d9c63d42a5a8\") " Jul 29 04:28:59.733707 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:59.733523 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qn4s6\" (UniqueName: \"kubernetes.io/projected/488c2811-20b2-4919-b601-d9c63d42a5a8-kube-api-access-qn4s6\") pod \"488c2811-20b2-4919-b601-d9c63d42a5a8\" (UID: \"488c2811-20b2-4919-b601-d9c63d42a5a8\") " Jul 29 04:28:59.733707 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:59.733573 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/488c2811-20b2-4919-b601-d9c63d42a5a8-dshm\") pod \"488c2811-20b2-4919-b601-d9c63d42a5a8\" (UID: \"488c2811-20b2-4919-b601-d9c63d42a5a8\") " Jul 29 04:28:59.733707 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:59.733592 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/488c2811-20b2-4919-b601-d9c63d42a5a8-tmp-dir\") pod \"488c2811-20b2-4919-b601-d9c63d42a5a8\" (UID: \"488c2811-20b2-4919-b601-d9c63d42a5a8\") " Jul 29 04:28:59.733707 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:59.733610 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/488c2811-20b2-4919-b601-d9c63d42a5a8-model-cache\") pod \"488c2811-20b2-4919-b601-d9c63d42a5a8\" (UID: \"488c2811-20b2-4919-b601-d9c63d42a5a8\") " Jul 29 04:28:59.733707 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:59.733659 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/488c2811-20b2-4919-b601-d9c63d42a5a8-home\") pod \"488c2811-20b2-4919-b601-d9c63d42a5a8\" (UID: \"488c2811-20b2-4919-b601-d9c63d42a5a8\") " Jul 29 04:28:59.734368 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:59.734310 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488c2811-20b2-4919-b601-d9c63d42a5a8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "488c2811-20b2-4919-b601-d9c63d42a5a8" (UID: "488c2811-20b2-4919-b601-d9c63d42a5a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 04:28:59.734537 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:59.734468 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488c2811-20b2-4919-b601-d9c63d42a5a8-model-cache" (OuterVolumeSpecName: "model-cache") pod "488c2811-20b2-4919-b601-d9c63d42a5a8" (UID: "488c2811-20b2-4919-b601-d9c63d42a5a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 04:28:59.734711 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:59.734670 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488c2811-20b2-4919-b601-d9c63d42a5a8-home" (OuterVolumeSpecName: "home") pod "488c2811-20b2-4919-b601-d9c63d42a5a8" (UID: "488c2811-20b2-4919-b601-d9c63d42a5a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 04:28:59.736033 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:59.735998 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488c2811-20b2-4919-b601-d9c63d42a5a8-dshm" (OuterVolumeSpecName: "dshm") pod "488c2811-20b2-4919-b601-d9c63d42a5a8" (UID: "488c2811-20b2-4919-b601-d9c63d42a5a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 04:28:59.736236 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:59.736191 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/488c2811-20b2-4919-b601-d9c63d42a5a8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "488c2811-20b2-4919-b601-d9c63d42a5a8" (UID: "488c2811-20b2-4919-b601-d9c63d42a5a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 04:28:59.736350 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:59.736279 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/488c2811-20b2-4919-b601-d9c63d42a5a8-kube-api-access-qn4s6" (OuterVolumeSpecName: "kube-api-access-qn4s6") pod "488c2811-20b2-4919-b601-d9c63d42a5a8" (UID: "488c2811-20b2-4919-b601-d9c63d42a5a8"). InnerVolumeSpecName "kube-api-access-qn4s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:28:59.834335 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:59.834299 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/488c2811-20b2-4919-b601-d9c63d42a5a8-home\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:28:59.834335 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:59.834326 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/488c2811-20b2-4919-b601-d9c63d42a5a8-tls-certs\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:28:59.834335 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:59.834338 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn4s6\" (UniqueName: \"kubernetes.io/projected/488c2811-20b2-4919-b601-d9c63d42a5a8-kube-api-access-qn4s6\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:28:59.834335 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:59.834347 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/488c2811-20b2-4919-b601-d9c63d42a5a8-dshm\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:28:59.834691 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:59.834375 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/488c2811-20b2-4919-b601-d9c63d42a5a8-tmp-dir\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:28:59.834691 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:28:59.834384 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/488c2811-20b2-4919-b601-d9c63d42a5a8-model-cache\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:29:00.574372 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:29:00.574341 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" Jul 29 04:29:00.574372 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:29:00.574351 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9" event={"ID":"488c2811-20b2-4919-b601-d9c63d42a5a8","Type":"ContainerDied","Data":"e34ab0b48060832e8fa7e5cdeaf0960386ffd868ea12b027311eff1ad68d17e5"} Jul 29 04:29:00.574804 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:29:00.574395 2579 scope.go:117] "RemoveContainer" containerID="dff877afe18018003a4beb8fb478dbaf79340d1002fc3b0be60971af18f66d21" Jul 29 04:29:00.594547 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:29:00.594522 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9"] Jul 29 04:29:00.597850 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:29:00.597823 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-c3af2471-kserve-65866f5767-tskd9"] Jul 29 04:29:01.591810 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:29:01.591728 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="488c2811-20b2-4919-b601-d9c63d42a5a8" path="/var/lib/kubelet/pods/488c2811-20b2-4919-b601-d9c63d42a5a8/volumes" Jul 29 04:34:01.125497 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:01.125461 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecf692c-d4pht"] Jul 29 04:34:01.126043 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:01.125839 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="488c2811-20b2-4919-b601-d9c63d42a5a8" containerName="main" Jul 29 04:34:01.126043 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:01.125850 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="488c2811-20b2-4919-b601-d9c63d42a5a8" containerName="main" Jul 29 04:34:01.126043 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:01.125921 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="488c2811-20b2-4919-b601-d9c63d42a5a8" containerName="main" Jul 29 04:34:01.128946 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:01.128923 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecf692c-d4pht" Jul 29 04:34:01.131159 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:01.131136 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mxv5w\"" Jul 29 04:34:01.135582 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:01.135556 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecf692c-d4pht"] Jul 29 04:34:01.194718 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:01.194675 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47243ef-2fb9-49ed-94b6-09c2e33831a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8ecf692c-d4pht\" (UID: \"a47243ef-2fb9-49ed-94b6-09c2e33831a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecf692c-d4pht" Jul 29 04:34:01.296012 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:01.295976 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47243ef-2fb9-49ed-94b6-09c2e33831a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8ecf692c-d4pht\" (UID: \"a47243ef-2fb9-49ed-94b6-09c2e33831a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecf692c-d4pht" Jul 29 04:34:01.296666 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:01.296647 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47243ef-2fb9-49ed-94b6-09c2e33831a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8ecf692c-d4pht\" (UID: \"a47243ef-2fb9-49ed-94b6-09c2e33831a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecf692c-d4pht" Jul 29 04:34:01.443467 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:01.443433 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecf692c-d4pht" Jul 29 04:34:01.578107 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:01.578085 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecf692c-d4pht"] Jul 29 04:34:01.580755 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:34:01.580725 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47243ef_2fb9_49ed_94b6_09c2e33831a9.slice/crio-f9df415fff3b47ee3c22395bfbe70e9ffafb71a12cc3b0b595c4e3fc4fcc7e29 WatchSource:0}: Error finding container f9df415fff3b47ee3c22395bfbe70e9ffafb71a12cc3b0b595c4e3fc4fcc7e29: Status 404 returned error can't find the container with id f9df415fff3b47ee3c22395bfbe70e9ffafb71a12cc3b0b595c4e3fc4fcc7e29 Jul 29 04:34:02.028331 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:02.028235 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecf692c-d4pht" event={"ID":"a47243ef-2fb9-49ed-94b6-09c2e33831a9","Type":"ContainerStarted","Data":"ad81846d95544a559cc049c23ec85f5cd0cfad3379369fc653763bbafb171aac"} Jul 29 04:34:02.028331 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:02.028305 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecf692c-d4pht" event={"ID":"a47243ef-2fb9-49ed-94b6-09c2e33831a9","Type":"ContainerStarted","Data":"f9df415fff3b47ee3c22395bfbe70e9ffafb71a12cc3b0b595c4e3fc4fcc7e29"} Jul 29 04:34:02.043756 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:02.043705 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecf692c-d4pht" podStartSLOduration=1.043689453 podStartE2EDuration="1.043689453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:34:02.041491545 +0000 UTC m=+1822.900866269" watchObservedRunningTime="2026-07-29 04:34:02.043689453 +0000 UTC m=+1822.903064177" Jul 29 04:34:03.033732 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:03.033689 2579 generic.go:356] "Generic (PLEG): container finished" podID="a47243ef-2fb9-49ed-94b6-09c2e33831a9" containerID="ad81846d95544a559cc049c23ec85f5cd0cfad3379369fc653763bbafb171aac" exitCode=0 Jul 29 04:34:03.034172 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:03.033783 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecf692c-d4pht" event={"ID":"a47243ef-2fb9-49ed-94b6-09c2e33831a9","Type":"ContainerDied","Data":"ad81846d95544a559cc049c23ec85f5cd0cfad3379369fc653763bbafb171aac"} Jul 29 04:34:04.177449 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:04.177422 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecf692c-d4pht" Jul 29 04:34:04.223407 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:04.223376 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47243ef-2fb9-49ed-94b6-09c2e33831a9-maas-api-service-ca\") pod \"a47243ef-2fb9-49ed-94b6-09c2e33831a9\" (UID: \"a47243ef-2fb9-49ed-94b6-09c2e33831a9\") " Jul 29 04:34:04.223719 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:04.223698 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a47243ef-2fb9-49ed-94b6-09c2e33831a9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a47243ef-2fb9-49ed-94b6-09c2e33831a9" (UID: "a47243ef-2fb9-49ed-94b6-09c2e33831a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 04:34:04.324249 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:04.324130 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47243ef-2fb9-49ed-94b6-09c2e33831a9-maas-api-service-ca\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:34:05.044300 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:05.044273 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecf692c-d4pht" Jul 29 04:34:05.044472 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:05.044294 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecf692c-d4pht" event={"ID":"a47243ef-2fb9-49ed-94b6-09c2e33831a9","Type":"ContainerDied","Data":"f9df415fff3b47ee3c22395bfbe70e9ffafb71a12cc3b0b595c4e3fc4fcc7e29"} Jul 29 04:34:05.044472 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:05.044343 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9df415fff3b47ee3c22395bfbe70e9ffafb71a12cc3b0b595c4e3fc4fcc7e29" Jul 29 04:34:06.439738 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:06.439704 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecf692c-d4pht"] Jul 29 04:34:06.442838 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:06.442814 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecf692c-d4pht"] Jul 29 04:34:07.594078 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:07.594040 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a47243ef-2fb9-49ed-94b6-09c2e33831a9" path="/var/lib/kubelet/pods/a47243ef-2fb9-49ed-94b6-09c2e33831a9/volumes" Jul 29 04:34:10.733814 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:10.733778 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a3bf19b0-6cd9b7559b-pxwbm"] Jul 29 04:34:10.734197 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:10.734153 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a47243ef-2fb9-49ed-94b6-09c2e33831a9" containerName="revoke-keys" Jul 29 04:34:10.734197 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:10.734165 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a47243ef-2fb9-49ed-94b6-09c2e33831a9" containerName="revoke-keys" Jul 29 04:34:10.734296 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:10.734246 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a47243ef-2fb9-49ed-94b6-09c2e33831a9" containerName="revoke-keys" Jul 29 04:34:10.737686 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:10.737665 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a3bf19b0-6cd9b7559b-pxwbm" Jul 29 04:34:10.740679 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:10.740656 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-a3bf19b0-dockercfg-598sv\"" Jul 29 04:34:10.740932 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:10.740915 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-a3bf19b0\"" Jul 29 04:34:10.748234 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:10.748179 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a3bf19b0-6cd9b7559b-pxwbm"] Jul 29 04:34:10.772290 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:10.772258 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a3bf19b0-bb4c4bdcc-dt64l"] Jul 29 04:34:10.775511 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:10.775495 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a3bf19b0-bb4c4bdcc-dt64l" Jul 29 04:34:10.784977 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:10.784954 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a3bf19b0-bb4c4bdcc-dt64l"] Jul 29 04:34:10.886764 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:10.886727 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45d5c571-68c6-44d5-93f8-ae7b45671039-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a3bf19b0-bb4c4bdcc-dt64l\" (UID: \"45d5c571-68c6-44d5-93f8-ae7b45671039\") " pod="openshift-ingress/payload-processing-e2e-migrate-a3bf19b0-bb4c4bdcc-dt64l" Jul 29 04:34:10.886764 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:10.886768 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pzcm\" (UniqueName: \"kubernetes.io/projected/45d5c571-68c6-44d5-93f8-ae7b45671039-kube-api-access-6pzcm\") pod \"payload-processing-e2e-migrate-a3bf19b0-bb4c4bdcc-dt64l\" (UID: \"45d5c571-68c6-44d5-93f8-ae7b45671039\") " pod="openshift-ingress/payload-processing-e2e-migrate-a3bf19b0-bb4c4bdcc-dt64l" Jul 29 04:34:10.886966 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:10.886833 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7565\" (UniqueName: \"kubernetes.io/projected/7e5e6742-7f1a-4ca6-9553-d72669388c66-kube-api-access-c7565\") pod \"payload-pre-processing-e2e-migrate-a3bf19b0-6cd9b7559b-pxwbm\" (UID: \"7e5e6742-7f1a-4ca6-9553-d72669388c66\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a3bf19b0-6cd9b7559b-pxwbm" Jul 29 04:34:10.886966 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:10.886915 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5e6742-7f1a-4ca6-9553-d72669388c66-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a3bf19b0-6cd9b7559b-pxwbm\" (UID: \"7e5e6742-7f1a-4ca6-9553-d72669388c66\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a3bf19b0-6cd9b7559b-pxwbm" Jul 29 04:34:10.988349 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:10.988258 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5e6742-7f1a-4ca6-9553-d72669388c66-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a3bf19b0-6cd9b7559b-pxwbm\" (UID: \"7e5e6742-7f1a-4ca6-9553-d72669388c66\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a3bf19b0-6cd9b7559b-pxwbm" Jul 29 04:34:10.988349 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:10.988303 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45d5c571-68c6-44d5-93f8-ae7b45671039-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a3bf19b0-bb4c4bdcc-dt64l\" (UID: \"45d5c571-68c6-44d5-93f8-ae7b45671039\") " pod="openshift-ingress/payload-processing-e2e-migrate-a3bf19b0-bb4c4bdcc-dt64l" Jul 29 04:34:10.988349 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:10.988325 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pzcm\" (UniqueName: \"kubernetes.io/projected/45d5c571-68c6-44d5-93f8-ae7b45671039-kube-api-access-6pzcm\") pod \"payload-processing-e2e-migrate-a3bf19b0-bb4c4bdcc-dt64l\" (UID: \"45d5c571-68c6-44d5-93f8-ae7b45671039\") " pod="openshift-ingress/payload-processing-e2e-migrate-a3bf19b0-bb4c4bdcc-dt64l" Jul 29 04:34:10.988652 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:10.988355 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7565\" (UniqueName: \"kubernetes.io/projected/7e5e6742-7f1a-4ca6-9553-d72669388c66-kube-api-access-c7565\") pod \"payload-pre-processing-e2e-migrate-a3bf19b0-6cd9b7559b-pxwbm\" (UID: \"7e5e6742-7f1a-4ca6-9553-d72669388c66\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a3bf19b0-6cd9b7559b-pxwbm" Jul 29 04:34:10.989127 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:10.989100 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5e6742-7f1a-4ca6-9553-d72669388c66-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a3bf19b0-6cd9b7559b-pxwbm\" (UID: \"7e5e6742-7f1a-4ca6-9553-d72669388c66\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a3bf19b0-6cd9b7559b-pxwbm" Jul 29 04:34:10.989254 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:10.989107 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45d5c571-68c6-44d5-93f8-ae7b45671039-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a3bf19b0-bb4c4bdcc-dt64l\" (UID: \"45d5c571-68c6-44d5-93f8-ae7b45671039\") " pod="openshift-ingress/payload-processing-e2e-migrate-a3bf19b0-bb4c4bdcc-dt64l" Jul 29 04:34:10.995819 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:10.995788 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7565\" (UniqueName: \"kubernetes.io/projected/7e5e6742-7f1a-4ca6-9553-d72669388c66-kube-api-access-c7565\") pod \"payload-pre-processing-e2e-migrate-a3bf19b0-6cd9b7559b-pxwbm\" (UID: \"7e5e6742-7f1a-4ca6-9553-d72669388c66\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a3bf19b0-6cd9b7559b-pxwbm" Jul 29 04:34:10.995949 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:10.995928 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pzcm\" (UniqueName: \"kubernetes.io/projected/45d5c571-68c6-44d5-93f8-ae7b45671039-kube-api-access-6pzcm\") pod \"payload-processing-e2e-migrate-a3bf19b0-bb4c4bdcc-dt64l\" (UID: \"45d5c571-68c6-44d5-93f8-ae7b45671039\") " pod="openshift-ingress/payload-processing-e2e-migrate-a3bf19b0-bb4c4bdcc-dt64l" Jul 29 04:34:11.048612 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:11.048577 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a3bf19b0-6cd9b7559b-pxwbm" Jul 29 04:34:11.086365 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:11.086331 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a3bf19b0-bb4c4bdcc-dt64l" Jul 29 04:34:11.199953 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:11.199918 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a3bf19b0-6cd9b7559b-pxwbm"] Jul 29 04:34:11.203018 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:34:11.202977 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5e6742_7f1a_4ca6_9553_d72669388c66.slice/crio-d14a9bdcd85759e5da2805feab459b89b924210ce80cbee170f17ec3eea31c85 WatchSource:0}: Error finding container d14a9bdcd85759e5da2805feab459b89b924210ce80cbee170f17ec3eea31c85: Status 404 returned error can't find the container with id d14a9bdcd85759e5da2805feab459b89b924210ce80cbee170f17ec3eea31c85 Jul 29 04:34:11.452141 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:11.452117 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a3bf19b0-bb4c4bdcc-dt64l"] Jul 29 04:34:11.454219 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:34:11.454178 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d5c571_68c6_44d5_93f8_ae7b45671039.slice/crio-fc11535fa8d36f20b0e6953a13bd51f192854369c4918f946321cfc89eb85ec0 WatchSource:0}: Error finding container fc11535fa8d36f20b0e6953a13bd51f192854369c4918f946321cfc89eb85ec0: Status 404 returned error can't find the container with id fc11535fa8d36f20b0e6953a13bd51f192854369c4918f946321cfc89eb85ec0 Jul 29 04:34:12.087677 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:12.087625 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a3bf19b0-6cd9b7559b-pxwbm" event={"ID":"7e5e6742-7f1a-4ca6-9553-d72669388c66","Type":"ContainerStarted","Data":"42437606be98e180ad97370b523a3d0f3698a81bf8f1dc2c8ce7fed8ff89bf96"} Jul 29 04:34:12.088134 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:12.087781 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a3bf19b0-6cd9b7559b-pxwbm" Jul 29 04:34:12.088134 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:12.087812 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a3bf19b0-6cd9b7559b-pxwbm" event={"ID":"7e5e6742-7f1a-4ca6-9553-d72669388c66","Type":"ContainerStarted","Data":"d14a9bdcd85759e5da2805feab459b89b924210ce80cbee170f17ec3eea31c85"} Jul 29 04:34:12.089093 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:12.089071 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a3bf19b0-bb4c4bdcc-dt64l" event={"ID":"45d5c571-68c6-44d5-93f8-ae7b45671039","Type":"ContainerStarted","Data":"730d0c6895c7eca8b3b1691664cdb57c7742b8a2b8ed7a7ed3a3963634b28058"} Jul 29 04:34:12.089175 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:12.089114 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a3bf19b0-bb4c4bdcc-dt64l" event={"ID":"45d5c571-68c6-44d5-93f8-ae7b45671039","Type":"ContainerStarted","Data":"fc11535fa8d36f20b0e6953a13bd51f192854369c4918f946321cfc89eb85ec0"} Jul 29 04:34:12.089175 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:12.089136 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-a3bf19b0-bb4c4bdcc-dt64l" Jul 29 04:34:12.102491 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:12.102447 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a3bf19b0-6cd9b7559b-pxwbm" podStartSLOduration=2.102434159 podStartE2EDuration="2.102434159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:34:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:34:12.101852415 +0000 UTC m=+1832.961227140" watchObservedRunningTime="2026-07-29 04:34:12.102434159 +0000 UTC m=+1832.961808883" Jul 29 04:34:12.116983 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:12.116938 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-a3bf19b0-bb4c4bdcc-dt64l" podStartSLOduration=2.116925148 podStartE2EDuration="2.116925148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:34:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:34:12.116037435 +0000 UTC m=+1832.975412159" watchObservedRunningTime="2026-07-29 04:34:12.116925148 +0000 UTC m=+1832.976299920" Jul 29 04:34:23.096598 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:23.096559 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-a3bf19b0-bb4c4bdcc-dt64l" Jul 29 04:34:23.097050 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:23.096636 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a3bf19b0-6cd9b7559b-pxwbm" Jul 29 04:34:40.572683 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:40.572648 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a3bf19b0-bb4c4bdcc-dt64l"] Jul 29 04:34:40.573303 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:40.572899 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-a3bf19b0-bb4c4bdcc-dt64l" podUID="45d5c571-68c6-44d5-93f8-ae7b45671039" containerName="payload-processing" containerID="cri-o://730d0c6895c7eca8b3b1691664cdb57c7742b8a2b8ed7a7ed3a3963634b28058" gracePeriod=30 Jul 29 04:34:40.578548 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:40.578516 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a3bf19b0-6cd9b7559b-pxwbm"] Jul 29 04:34:40.578829 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:40.578758 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a3bf19b0-6cd9b7559b-pxwbm" podUID="7e5e6742-7f1a-4ca6-9553-d72669388c66" containerName="payload-pre-processing" containerID="cri-o://42437606be98e180ad97370b523a3d0f3698a81bf8f1dc2c8ce7fed8ff89bf96" gracePeriod=30 Jul 29 04:34:40.631362 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:34:40.631325 2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5e6742_7f1a_4ca6_9553_d72669388c66.slice/crio-42437606be98e180ad97370b523a3d0f3698a81bf8f1dc2c8ce7fed8ff89bf96.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5e6742_7f1a_4ca6_9553_d72669388c66.slice/crio-conmon-42437606be98e180ad97370b523a3d0f3698a81bf8f1dc2c8ce7fed8ff89bf96.scope\": RecentStats: unable to find data in memory cache]" Jul 29 04:34:40.631561 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:34:40.631395 2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5e6742_7f1a_4ca6_9553_d72669388c66.slice/crio-conmon-42437606be98e180ad97370b523a3d0f3698a81bf8f1dc2c8ce7fed8ff89bf96.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5e6742_7f1a_4ca6_9553_d72669388c66.slice/crio-42437606be98e180ad97370b523a3d0f3698a81bf8f1dc2c8ce7fed8ff89bf96.scope\": RecentStats: unable to find data in memory cache]" Jul 29 04:34:40.841870 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:40.841520 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a3bf19b0-bb4c4bdcc-dt64l" Jul 29 04:34:40.879101 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:40.879075 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a3bf19b0-6cd9b7559b-pxwbm" Jul 29 04:34:40.937513 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:40.937473 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pzcm\" (UniqueName: \"kubernetes.io/projected/45d5c571-68c6-44d5-93f8-ae7b45671039-kube-api-access-6pzcm\") pod \"45d5c571-68c6-44d5-93f8-ae7b45671039\" (UID: \"45d5c571-68c6-44d5-93f8-ae7b45671039\") " Jul 29 04:34:40.937702 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:40.937568 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7565\" (UniqueName: \"kubernetes.io/projected/7e5e6742-7f1a-4ca6-9553-d72669388c66-kube-api-access-c7565\") pod \"7e5e6742-7f1a-4ca6-9553-d72669388c66\" (UID: \"7e5e6742-7f1a-4ca6-9553-d72669388c66\") " Jul 29 04:34:40.937702 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:40.937633 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45d5c571-68c6-44d5-93f8-ae7b45671039-plugins-config-volume\") pod \"45d5c571-68c6-44d5-93f8-ae7b45671039\" (UID: \"45d5c571-68c6-44d5-93f8-ae7b45671039\") " Jul 29 04:34:40.938050 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:40.938018 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45d5c571-68c6-44d5-93f8-ae7b45671039-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "45d5c571-68c6-44d5-93f8-ae7b45671039" (UID: "45d5c571-68c6-44d5-93f8-ae7b45671039"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 04:34:40.938276 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:40.938090 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5e6742-7f1a-4ca6-9553-d72669388c66-plugins-config-volume\") pod \"7e5e6742-7f1a-4ca6-9553-d72669388c66\" (UID: \"7e5e6742-7f1a-4ca6-9553-d72669388c66\") " Jul 29 04:34:40.938541 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:40.938517 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e5e6742-7f1a-4ca6-9553-d72669388c66-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7e5e6742-7f1a-4ca6-9553-d72669388c66" (UID: "7e5e6742-7f1a-4ca6-9553-d72669388c66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 04:34:40.938803 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:40.938785 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45d5c571-68c6-44d5-93f8-ae7b45671039-plugins-config-volume\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:34:40.938868 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:40.938808 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5e6742-7f1a-4ca6-9553-d72669388c66-plugins-config-volume\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:34:40.940167 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:40.940136 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e5e6742-7f1a-4ca6-9553-d72669388c66-kube-api-access-c7565" (OuterVolumeSpecName: "kube-api-access-c7565") pod "7e5e6742-7f1a-4ca6-9553-d72669388c66" (UID: "7e5e6742-7f1a-4ca6-9553-d72669388c66"). InnerVolumeSpecName "kube-api-access-c7565". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:34:40.940167 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:40.940151 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d5c571-68c6-44d5-93f8-ae7b45671039-kube-api-access-6pzcm" (OuterVolumeSpecName: "kube-api-access-6pzcm") pod "45d5c571-68c6-44d5-93f8-ae7b45671039" (UID: "45d5c571-68c6-44d5-93f8-ae7b45671039"). InnerVolumeSpecName "kube-api-access-6pzcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:34:41.040160 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:41.040128 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pzcm\" (UniqueName: \"kubernetes.io/projected/45d5c571-68c6-44d5-93f8-ae7b45671039-kube-api-access-6pzcm\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:34:41.040160 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:41.040155 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7565\" (UniqueName: \"kubernetes.io/projected/7e5e6742-7f1a-4ca6-9553-d72669388c66-kube-api-access-c7565\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:34:41.233380 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:41.233346 2579 generic.go:356] "Generic (PLEG): container finished" podID="7e5e6742-7f1a-4ca6-9553-d72669388c66" containerID="42437606be98e180ad97370b523a3d0f3698a81bf8f1dc2c8ce7fed8ff89bf96" exitCode=0 Jul 29 04:34:41.233573 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:41.233387 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a3bf19b0-6cd9b7559b-pxwbm" Jul 29 04:34:41.233573 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:41.233429 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a3bf19b0-6cd9b7559b-pxwbm" event={"ID":"7e5e6742-7f1a-4ca6-9553-d72669388c66","Type":"ContainerDied","Data":"42437606be98e180ad97370b523a3d0f3698a81bf8f1dc2c8ce7fed8ff89bf96"} Jul 29 04:34:41.233573 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:41.233482 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a3bf19b0-6cd9b7559b-pxwbm" event={"ID":"7e5e6742-7f1a-4ca6-9553-d72669388c66","Type":"ContainerDied","Data":"d14a9bdcd85759e5da2805feab459b89b924210ce80cbee170f17ec3eea31c85"} Jul 29 04:34:41.233573 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:41.233505 2579 scope.go:117] "RemoveContainer" containerID="42437606be98e180ad97370b523a3d0f3698a81bf8f1dc2c8ce7fed8ff89bf96" Jul 29 04:34:41.234905 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:41.234879 2579 generic.go:356] "Generic (PLEG): container finished" podID="45d5c571-68c6-44d5-93f8-ae7b45671039" containerID="730d0c6895c7eca8b3b1691664cdb57c7742b8a2b8ed7a7ed3a3963634b28058" exitCode=0 Jul 29 04:34:41.235040 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:41.234921 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a3bf19b0-bb4c4bdcc-dt64l" Jul 29 04:34:41.235040 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:41.234967 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a3bf19b0-bb4c4bdcc-dt64l" event={"ID":"45d5c571-68c6-44d5-93f8-ae7b45671039","Type":"ContainerDied","Data":"730d0c6895c7eca8b3b1691664cdb57c7742b8a2b8ed7a7ed3a3963634b28058"} Jul 29 04:34:41.235040 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:41.235008 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a3bf19b0-bb4c4bdcc-dt64l" event={"ID":"45d5c571-68c6-44d5-93f8-ae7b45671039","Type":"ContainerDied","Data":"fc11535fa8d36f20b0e6953a13bd51f192854369c4918f946321cfc89eb85ec0"} Jul 29 04:34:41.245948 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:41.245807 2579 scope.go:117] "RemoveContainer" containerID="42437606be98e180ad97370b523a3d0f3698a81bf8f1dc2c8ce7fed8ff89bf96" Jul 29 04:34:41.249150 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:34:41.247294 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42437606be98e180ad97370b523a3d0f3698a81bf8f1dc2c8ce7fed8ff89bf96\": container with ID starting with 42437606be98e180ad97370b523a3d0f3698a81bf8f1dc2c8ce7fed8ff89bf96 not found: ID does not exist" containerID="42437606be98e180ad97370b523a3d0f3698a81bf8f1dc2c8ce7fed8ff89bf96" Jul 29 04:34:41.249150 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:41.247340 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42437606be98e180ad97370b523a3d0f3698a81bf8f1dc2c8ce7fed8ff89bf96"} err="failed to get container status \"42437606be98e180ad97370b523a3d0f3698a81bf8f1dc2c8ce7fed8ff89bf96\": rpc error: code = NotFound desc = could not find container \"42437606be98e180ad97370b523a3d0f3698a81bf8f1dc2c8ce7fed8ff89bf96\": container with ID starting with 42437606be98e180ad97370b523a3d0f3698a81bf8f1dc2c8ce7fed8ff89bf96 not found: ID does not exist" Jul 29 04:34:41.249150 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:41.247365 2579 scope.go:117] "RemoveContainer" containerID="730d0c6895c7eca8b3b1691664cdb57c7742b8a2b8ed7a7ed3a3963634b28058" Jul 29 04:34:41.256906 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:41.256885 2579 scope.go:117] "RemoveContainer" containerID="730d0c6895c7eca8b3b1691664cdb57c7742b8a2b8ed7a7ed3a3963634b28058" Jul 29 04:34:41.257237 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:34:41.257200 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"730d0c6895c7eca8b3b1691664cdb57c7742b8a2b8ed7a7ed3a3963634b28058\": container with ID starting with 730d0c6895c7eca8b3b1691664cdb57c7742b8a2b8ed7a7ed3a3963634b28058 not found: ID does not exist" containerID="730d0c6895c7eca8b3b1691664cdb57c7742b8a2b8ed7a7ed3a3963634b28058" Jul 29 04:34:41.257317 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:41.257244 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"730d0c6895c7eca8b3b1691664cdb57c7742b8a2b8ed7a7ed3a3963634b28058"} err="failed to get container status \"730d0c6895c7eca8b3b1691664cdb57c7742b8a2b8ed7a7ed3a3963634b28058\": rpc error: code = NotFound desc = could not find container \"730d0c6895c7eca8b3b1691664cdb57c7742b8a2b8ed7a7ed3a3963634b28058\": container with ID starting with 730d0c6895c7eca8b3b1691664cdb57c7742b8a2b8ed7a7ed3a3963634b28058 not found: ID does not exist" Jul 29 04:34:41.261392 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:41.261369 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a3bf19b0-6cd9b7559b-pxwbm"] Jul 29 04:34:41.263219 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:41.263182 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a3bf19b0-6cd9b7559b-pxwbm"] Jul 29 04:34:41.272861 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:41.272839 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a3bf19b0-bb4c4bdcc-dt64l"] Jul 29 04:34:41.276969 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:41.276950 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a3bf19b0-bb4c4bdcc-dt64l"] Jul 29 04:34:41.594629 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:41.594595 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45d5c571-68c6-44d5-93f8-ae7b45671039" path="/var/lib/kubelet/pods/45d5c571-68c6-44d5-93f8-ae7b45671039/volumes" Jul 29 04:34:41.595155 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:41.595134 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e5e6742-7f1a-4ca6-9553-d72669388c66" path="/var/lib/kubelet/pods/7e5e6742-7f1a-4ca6-9553-d72669388c66/volumes" Jul 29 04:34:44.740356 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:44.740323 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0c9d218b-cc8c8db8d-htlkt"] Jul 29 04:34:44.740832 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:44.740782 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e5e6742-7f1a-4ca6-9553-d72669388c66" containerName="payload-pre-processing" Jul 29 04:34:44.740832 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:44.740800 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e5e6742-7f1a-4ca6-9553-d72669388c66" containerName="payload-pre-processing" Jul 29 04:34:44.740832 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:44.740819 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45d5c571-68c6-44d5-93f8-ae7b45671039" containerName="payload-processing" Jul 29 04:34:44.740832 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:44.740829 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="45d5c571-68c6-44d5-93f8-ae7b45671039" containerName="payload-processing" Jul 29 04:34:44.741079 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:44.740915 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e5e6742-7f1a-4ca6-9553-d72669388c66" containerName="payload-pre-processing" Jul 29 04:34:44.741079 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:44.740928 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="45d5c571-68c6-44d5-93f8-ae7b45671039" containerName="payload-processing" Jul 29 04:34:44.743923 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:44.743906 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0c9d218b-cc8c8db8d-htlkt" Jul 29 04:34:44.746288 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:44.746267 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0c9d218b\"" Jul 29 04:34:44.746359 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:44.746267 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0c9d218b-dockercfg-4fc65\"" Jul 29 04:34:44.753847 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:44.753826 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0c9d218b-cc8c8db8d-htlkt"] Jul 29 04:34:44.772833 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:44.772796 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0c9d218b-69dccd6474-thwwz"] Jul 29 04:34:44.776519 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:44.776495 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0c9d218b-69dccd6474-thwwz" Jul 29 04:34:44.778740 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:44.778713 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8448f20-b8be-45ea-bba0-5a44bd8da783-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0c9d218b-cc8c8db8d-htlkt\" (UID: \"b8448f20-b8be-45ea-bba0-5a44bd8da783\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0c9d218b-cc8c8db8d-htlkt" Jul 29 04:34:44.778878 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:44.778814 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvf2m\" (UniqueName: \"kubernetes.io/projected/b8448f20-b8be-45ea-bba0-5a44bd8da783-kube-api-access-dvf2m\") pod \"payload-pre-processing-e2e-ait-0c9d218b-cc8c8db8d-htlkt\" (UID: \"b8448f20-b8be-45ea-bba0-5a44bd8da783\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0c9d218b-cc8c8db8d-htlkt" Jul 29 04:34:44.784564 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:44.784518 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0c9d218b-69dccd6474-thwwz"] Jul 29 04:34:44.879444 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:44.879413 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvf2m\" (UniqueName: \"kubernetes.io/projected/b8448f20-b8be-45ea-bba0-5a44bd8da783-kube-api-access-dvf2m\") pod \"payload-pre-processing-e2e-ait-0c9d218b-cc8c8db8d-htlkt\" (UID: \"b8448f20-b8be-45ea-bba0-5a44bd8da783\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0c9d218b-cc8c8db8d-htlkt" Jul 29 04:34:44.879648 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:44.879454 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8448f20-b8be-45ea-bba0-5a44bd8da783-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0c9d218b-cc8c8db8d-htlkt\" (UID: \"b8448f20-b8be-45ea-bba0-5a44bd8da783\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0c9d218b-cc8c8db8d-htlkt" Jul 29 04:34:44.879648 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:44.879478 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8fc21d3-7197-4477-a6c8-e349b11257c3-plugins-config-volume\") pod \"payload-processing-e2e-ait-0c9d218b-69dccd6474-thwwz\" (UID: \"a8fc21d3-7197-4477-a6c8-e349b11257c3\") " pod="openshift-ingress/payload-processing-e2e-ait-0c9d218b-69dccd6474-thwwz" Jul 29 04:34:44.879648 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:44.879541 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jx46\" (UniqueName: \"kubernetes.io/projected/a8fc21d3-7197-4477-a6c8-e349b11257c3-kube-api-access-7jx46\") pod \"payload-processing-e2e-ait-0c9d218b-69dccd6474-thwwz\" (UID: \"a8fc21d3-7197-4477-a6c8-e349b11257c3\") " pod="openshift-ingress/payload-processing-e2e-ait-0c9d218b-69dccd6474-thwwz" Jul 29 04:34:44.880401 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:44.880380 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8448f20-b8be-45ea-bba0-5a44bd8da783-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0c9d218b-cc8c8db8d-htlkt\" (UID: \"b8448f20-b8be-45ea-bba0-5a44bd8da783\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0c9d218b-cc8c8db8d-htlkt" Jul 29 04:34:44.887324 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:44.887299 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvf2m\" (UniqueName: \"kubernetes.io/projected/b8448f20-b8be-45ea-bba0-5a44bd8da783-kube-api-access-dvf2m\") pod \"payload-pre-processing-e2e-ait-0c9d218b-cc8c8db8d-htlkt\" (UID: \"b8448f20-b8be-45ea-bba0-5a44bd8da783\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0c9d218b-cc8c8db8d-htlkt" Jul 29 04:34:44.980304 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:44.980267 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8fc21d3-7197-4477-a6c8-e349b11257c3-plugins-config-volume\") pod \"payload-processing-e2e-ait-0c9d218b-69dccd6474-thwwz\" (UID: \"a8fc21d3-7197-4477-a6c8-e349b11257c3\") " pod="openshift-ingress/payload-processing-e2e-ait-0c9d218b-69dccd6474-thwwz" Jul 29 04:34:44.980485 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:44.980359 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jx46\" (UniqueName: \"kubernetes.io/projected/a8fc21d3-7197-4477-a6c8-e349b11257c3-kube-api-access-7jx46\") pod \"payload-processing-e2e-ait-0c9d218b-69dccd6474-thwwz\" (UID: \"a8fc21d3-7197-4477-a6c8-e349b11257c3\") " pod="openshift-ingress/payload-processing-e2e-ait-0c9d218b-69dccd6474-thwwz" Jul 29 04:34:44.980933 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:44.980911 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8fc21d3-7197-4477-a6c8-e349b11257c3-plugins-config-volume\") pod \"payload-processing-e2e-ait-0c9d218b-69dccd6474-thwwz\" (UID: \"a8fc21d3-7197-4477-a6c8-e349b11257c3\") " pod="openshift-ingress/payload-processing-e2e-ait-0c9d218b-69dccd6474-thwwz" Jul 29 04:34:44.987315 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:44.987293 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jx46\" (UniqueName: \"kubernetes.io/projected/a8fc21d3-7197-4477-a6c8-e349b11257c3-kube-api-access-7jx46\") pod \"payload-processing-e2e-ait-0c9d218b-69dccd6474-thwwz\" (UID: \"a8fc21d3-7197-4477-a6c8-e349b11257c3\") " pod="openshift-ingress/payload-processing-e2e-ait-0c9d218b-69dccd6474-thwwz" Jul 29 04:34:45.055878 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:45.055789 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0c9d218b-cc8c8db8d-htlkt" Jul 29 04:34:45.088786 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:45.088758 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0c9d218b-69dccd6474-thwwz" Jul 29 04:34:45.206319 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:45.206286 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0c9d218b-cc8c8db8d-htlkt"] Jul 29 04:34:45.207999 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:34:45.207970 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8448f20_b8be_45ea_bba0_5a44bd8da783.slice/crio-cfb228a03d331272250e2955e6b98a1c8fa5c015d6de69edd6324d3f01530e27 WatchSource:0}: Error finding container cfb228a03d331272250e2955e6b98a1c8fa5c015d6de69edd6324d3f01530e27: Status 404 returned error can't find the container with id cfb228a03d331272250e2955e6b98a1c8fa5c015d6de69edd6324d3f01530e27 Jul 29 04:34:45.249526 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:45.249329 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0c9d218b-69dccd6474-thwwz"] Jul 29 04:34:45.251659 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:34:45.251633 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8fc21d3_7197_4477_a6c8_e349b11257c3.slice/crio-215df5b5ee0a914002978d06c972fa4d9cefe1f619ced21d489057cf6b31de53 WatchSource:0}: Error finding container 215df5b5ee0a914002978d06c972fa4d9cefe1f619ced21d489057cf6b31de53: Status 404 returned error can't find the container with id 215df5b5ee0a914002978d06c972fa4d9cefe1f619ced21d489057cf6b31de53 Jul 29 04:34:45.259683 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:45.259660 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0c9d218b-cc8c8db8d-htlkt" event={"ID":"b8448f20-b8be-45ea-bba0-5a44bd8da783","Type":"ContainerStarted","Data":"cfb228a03d331272250e2955e6b98a1c8fa5c015d6de69edd6324d3f01530e27"} Jul 29 04:34:45.260629 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:45.260608 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0c9d218b-69dccd6474-thwwz" event={"ID":"a8fc21d3-7197-4477-a6c8-e349b11257c3","Type":"ContainerStarted","Data":"215df5b5ee0a914002978d06c972fa4d9cefe1f619ced21d489057cf6b31de53"} Jul 29 04:34:46.267057 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:46.267017 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0c9d218b-cc8c8db8d-htlkt" event={"ID":"b8448f20-b8be-45ea-bba0-5a44bd8da783","Type":"ContainerStarted","Data":"9cc6766f1b0d325b3c9f9e7b023fd69ae113b75b8311901586b9db8b1637dc82"} Jul 29 04:34:46.267685 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:46.267184 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0c9d218b-cc8c8db8d-htlkt" Jul 29 04:34:46.268605 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:46.268579 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0c9d218b-69dccd6474-thwwz" event={"ID":"a8fc21d3-7197-4477-a6c8-e349b11257c3","Type":"ContainerStarted","Data":"ba70c289bd74c13f104bd87abfb4f58ceb57727b7ab44292e4183d0fe577d724"} Jul 29 04:34:46.268712 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:46.268633 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0c9d218b-69dccd6474-thwwz" Jul 29 04:34:46.281279 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:46.281236 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0c9d218b-cc8c8db8d-htlkt" podStartSLOduration=2.281194496 podStartE2EDuration="2.281194496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:34:46.28025915 +0000 UTC m=+1867.139633874" watchObservedRunningTime="2026-07-29 04:34:46.281194496 +0000 UTC m=+1867.140569219" Jul 29 04:34:46.297814 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:46.297769 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0c9d218b-69dccd6474-thwwz" podStartSLOduration=2.2977555880000002 podStartE2EDuration="2.297755588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:34:46.295687435 +0000 UTC m=+1867.155062171" watchObservedRunningTime="2026-07-29 04:34:46.297755588 +0000 UTC m=+1867.157130356" Jul 29 04:34:57.275121 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:57.275092 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0c9d218b-cc8c8db8d-htlkt" Jul 29 04:34:57.275591 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:34:57.275277 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0c9d218b-69dccd6474-thwwz" Jul 29 04:35:14.709431 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:14.709342 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0c9d218b-69dccd6474-thwwz"] Jul 29 04:35:14.709989 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:14.709598 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0c9d218b-69dccd6474-thwwz" podUID="a8fc21d3-7197-4477-a6c8-e349b11257c3" containerName="payload-processing" containerID="cri-o://ba70c289bd74c13f104bd87abfb4f58ceb57727b7ab44292e4183d0fe577d724" gracePeriod=30 Jul 29 04:35:14.716105 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:14.716067 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0c9d218b-cc8c8db8d-htlkt"] Jul 29 04:35:14.716332 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:14.716305 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0c9d218b-cc8c8db8d-htlkt" podUID="b8448f20-b8be-45ea-bba0-5a44bd8da783" containerName="payload-pre-processing" containerID="cri-o://9cc6766f1b0d325b3c9f9e7b023fd69ae113b75b8311901586b9db8b1637dc82" gracePeriod=30 Jul 29 04:35:14.987174 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:14.987150 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0c9d218b-69dccd6474-thwwz" Jul 29 04:35:15.000925 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.000899 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0c9d218b-cc8c8db8d-htlkt" Jul 29 04:35:15.154685 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.154653 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvf2m\" (UniqueName: \"kubernetes.io/projected/b8448f20-b8be-45ea-bba0-5a44bd8da783-kube-api-access-dvf2m\") pod \"b8448f20-b8be-45ea-bba0-5a44bd8da783\" (UID: \"b8448f20-b8be-45ea-bba0-5a44bd8da783\") " Jul 29 04:35:15.154877 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.154743 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8448f20-b8be-45ea-bba0-5a44bd8da783-plugins-config-volume\") pod \"b8448f20-b8be-45ea-bba0-5a44bd8da783\" (UID: \"b8448f20-b8be-45ea-bba0-5a44bd8da783\") " Jul 29 04:35:15.154877 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.154773 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8fc21d3-7197-4477-a6c8-e349b11257c3-plugins-config-volume\") pod \"a8fc21d3-7197-4477-a6c8-e349b11257c3\" (UID: \"a8fc21d3-7197-4477-a6c8-e349b11257c3\") " Jul 29 04:35:15.154877 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.154804 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jx46\" (UniqueName: \"kubernetes.io/projected/a8fc21d3-7197-4477-a6c8-e349b11257c3-kube-api-access-7jx46\") pod \"a8fc21d3-7197-4477-a6c8-e349b11257c3\" (UID: \"a8fc21d3-7197-4477-a6c8-e349b11257c3\") " Jul 29 04:35:15.155176 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.155145 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8448f20-b8be-45ea-bba0-5a44bd8da783-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b8448f20-b8be-45ea-bba0-5a44bd8da783" (UID: "b8448f20-b8be-45ea-bba0-5a44bd8da783"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 04:35:15.155176 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.155164 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8fc21d3-7197-4477-a6c8-e349b11257c3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a8fc21d3-7197-4477-a6c8-e349b11257c3" (UID: "a8fc21d3-7197-4477-a6c8-e349b11257c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 04:35:15.156830 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.156806 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8448f20-b8be-45ea-bba0-5a44bd8da783-kube-api-access-dvf2m" (OuterVolumeSpecName: "kube-api-access-dvf2m") pod "b8448f20-b8be-45ea-bba0-5a44bd8da783" (UID: "b8448f20-b8be-45ea-bba0-5a44bd8da783"). InnerVolumeSpecName "kube-api-access-dvf2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:35:15.156915 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.156864 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8fc21d3-7197-4477-a6c8-e349b11257c3-kube-api-access-7jx46" (OuterVolumeSpecName: "kube-api-access-7jx46") pod "a8fc21d3-7197-4477-a6c8-e349b11257c3" (UID: "a8fc21d3-7197-4477-a6c8-e349b11257c3"). InnerVolumeSpecName "kube-api-access-7jx46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:35:15.255906 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.255820 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8448f20-b8be-45ea-bba0-5a44bd8da783-plugins-config-volume\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:35:15.255906 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.255849 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8fc21d3-7197-4477-a6c8-e349b11257c3-plugins-config-volume\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:35:15.255906 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.255858 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jx46\" (UniqueName: \"kubernetes.io/projected/a8fc21d3-7197-4477-a6c8-e349b11257c3-kube-api-access-7jx46\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:35:15.255906 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.255869 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvf2m\" (UniqueName: \"kubernetes.io/projected/b8448f20-b8be-45ea-bba0-5a44bd8da783-kube-api-access-dvf2m\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:35:15.416790 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.416752 2579 generic.go:356] "Generic (PLEG): container finished" podID="b8448f20-b8be-45ea-bba0-5a44bd8da783" containerID="9cc6766f1b0d325b3c9f9e7b023fd69ae113b75b8311901586b9db8b1637dc82" exitCode=0 Jul 29 04:35:15.416990 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.416801 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0c9d218b-cc8c8db8d-htlkt" Jul 29 04:35:15.416990 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.416835 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0c9d218b-cc8c8db8d-htlkt" event={"ID":"b8448f20-b8be-45ea-bba0-5a44bd8da783","Type":"ContainerDied","Data":"9cc6766f1b0d325b3c9f9e7b023fd69ae113b75b8311901586b9db8b1637dc82"} Jul 29 04:35:15.416990 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.416900 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0c9d218b-cc8c8db8d-htlkt" event={"ID":"b8448f20-b8be-45ea-bba0-5a44bd8da783","Type":"ContainerDied","Data":"cfb228a03d331272250e2955e6b98a1c8fa5c015d6de69edd6324d3f01530e27"} Jul 29 04:35:15.416990 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.416929 2579 scope.go:117] "RemoveContainer" containerID="9cc6766f1b0d325b3c9f9e7b023fd69ae113b75b8311901586b9db8b1637dc82" Jul 29 04:35:15.418202 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.418179 2579 generic.go:356] "Generic (PLEG): container finished" podID="a8fc21d3-7197-4477-a6c8-e349b11257c3" containerID="ba70c289bd74c13f104bd87abfb4f58ceb57727b7ab44292e4183d0fe577d724" exitCode=0 Jul 29 04:35:15.418320 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.418238 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0c9d218b-69dccd6474-thwwz" Jul 29 04:35:15.418320 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.418270 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0c9d218b-69dccd6474-thwwz" event={"ID":"a8fc21d3-7197-4477-a6c8-e349b11257c3","Type":"ContainerDied","Data":"ba70c289bd74c13f104bd87abfb4f58ceb57727b7ab44292e4183d0fe577d724"} Jul 29 04:35:15.418320 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.418313 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0c9d218b-69dccd6474-thwwz" event={"ID":"a8fc21d3-7197-4477-a6c8-e349b11257c3","Type":"ContainerDied","Data":"215df5b5ee0a914002978d06c972fa4d9cefe1f619ced21d489057cf6b31de53"} Jul 29 04:35:15.427161 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.427143 2579 scope.go:117] "RemoveContainer" containerID="9cc6766f1b0d325b3c9f9e7b023fd69ae113b75b8311901586b9db8b1637dc82" Jul 29 04:35:15.427715 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:35:15.427693 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cc6766f1b0d325b3c9f9e7b023fd69ae113b75b8311901586b9db8b1637dc82\": container with ID starting with 9cc6766f1b0d325b3c9f9e7b023fd69ae113b75b8311901586b9db8b1637dc82 not found: ID does not exist" containerID="9cc6766f1b0d325b3c9f9e7b023fd69ae113b75b8311901586b9db8b1637dc82" Jul 29 04:35:15.427826 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.427721 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cc6766f1b0d325b3c9f9e7b023fd69ae113b75b8311901586b9db8b1637dc82"} err="failed to get container status \"9cc6766f1b0d325b3c9f9e7b023fd69ae113b75b8311901586b9db8b1637dc82\": rpc error: code = NotFound desc = could not find container \"9cc6766f1b0d325b3c9f9e7b023fd69ae113b75b8311901586b9db8b1637dc82\": container with ID starting with 9cc6766f1b0d325b3c9f9e7b023fd69ae113b75b8311901586b9db8b1637dc82 not found: ID does not exist" Jul 29 04:35:15.427826 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.427737 2579 scope.go:117] "RemoveContainer" containerID="ba70c289bd74c13f104bd87abfb4f58ceb57727b7ab44292e4183d0fe577d724" Jul 29 04:35:15.437000 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.436984 2579 scope.go:117] "RemoveContainer" containerID="ba70c289bd74c13f104bd87abfb4f58ceb57727b7ab44292e4183d0fe577d724" Jul 29 04:35:15.437262 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:35:15.437243 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba70c289bd74c13f104bd87abfb4f58ceb57727b7ab44292e4183d0fe577d724\": container with ID starting with ba70c289bd74c13f104bd87abfb4f58ceb57727b7ab44292e4183d0fe577d724 not found: ID does not exist" containerID="ba70c289bd74c13f104bd87abfb4f58ceb57727b7ab44292e4183d0fe577d724" Jul 29 04:35:15.437341 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.437273 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba70c289bd74c13f104bd87abfb4f58ceb57727b7ab44292e4183d0fe577d724"} err="failed to get container status \"ba70c289bd74c13f104bd87abfb4f58ceb57727b7ab44292e4183d0fe577d724\": rpc error: code = NotFound desc = could not find container \"ba70c289bd74c13f104bd87abfb4f58ceb57727b7ab44292e4183d0fe577d724\": container with ID starting with ba70c289bd74c13f104bd87abfb4f58ceb57727b7ab44292e4183d0fe577d724 not found: ID does not exist" Jul 29 04:35:15.442753 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.442733 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0c9d218b-cc8c8db8d-htlkt"] Jul 29 04:35:15.445942 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.445922 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0c9d218b-cc8c8db8d-htlkt"] Jul 29 04:35:15.455520 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.455472 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0c9d218b-69dccd6474-thwwz"] Jul 29 04:35:15.457006 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.456988 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0c9d218b-69dccd6474-thwwz"] Jul 29 04:35:15.592377 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.592290 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8fc21d3-7197-4477-a6c8-e349b11257c3" path="/var/lib/kubelet/pods/a8fc21d3-7197-4477-a6c8-e349b11257c3/volumes" Jul 29 04:35:15.592654 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:15.592641 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8448f20-b8be-45ea-bba0-5a44bd8da783" path="/var/lib/kubelet/pods/b8448f20-b8be-45ea-bba0-5a44bd8da783/volumes" Jul 29 04:35:20.025927 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:20.025899 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-73de3e0f-544dd76cf6-hjzm7"] Jul 29 04:35:20.026440 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:20.026327 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8fc21d3-7197-4477-a6c8-e349b11257c3" containerName="payload-processing" Jul 29 04:35:20.026440 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:20.026339 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8fc21d3-7197-4477-a6c8-e349b11257c3" containerName="payload-processing" Jul 29 04:35:20.026440 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:20.026355 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8448f20-b8be-45ea-bba0-5a44bd8da783" containerName="payload-pre-processing" Jul 29 04:35:20.026440 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:20.026362 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8448f20-b8be-45ea-bba0-5a44bd8da783" containerName="payload-pre-processing" Jul 29 04:35:20.026440 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:20.026433 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8448f20-b8be-45ea-bba0-5a44bd8da783" containerName="payload-pre-processing" Jul 29 04:35:20.026440 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:20.026441 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8fc21d3-7197-4477-a6c8-e349b11257c3" containerName="payload-processing" Jul 29 04:35:20.031110 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:20.031085 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-73de3e0f-544dd76cf6-hjzm7" Jul 29 04:35:20.033243 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:20.033221 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-73de3e0f\"" Jul 29 04:35:20.039756 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:20.039732 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-73de3e0f-544dd76cf6-hjzm7"] Jul 29 04:35:20.064109 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:20.064077 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-73de3e0f-584b9745cd-mljwh"] Jul 29 04:35:20.067436 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:20.067421 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-73de3e0f-584b9745cd-mljwh" Jul 29 04:35:20.069608 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:20.069585 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-73de3e0f-dockercfg-gpckf\"" Jul 29 04:35:20.077074 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:20.077049 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-73de3e0f-584b9745cd-mljwh"] Jul 29 04:35:20.196322 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:20.196281 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plvnl\" (UniqueName: \"kubernetes.io/projected/347ecb1d-a89c-49c6-a7b8-70baf108c503-kube-api-access-plvnl\") pod \"payload-pre-processing-e2e-derive-73de3e0f-544dd76cf6-hjzm7\" (UID: \"347ecb1d-a89c-49c6-a7b8-70baf108c503\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-73de3e0f-544dd76cf6-hjzm7" Jul 29 04:35:20.196534 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:20.196375 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac1fdc03-4f97-45a8-83d7-83a881b9e483-plugins-config-volume\") pod \"payload-processing-e2e-derive-73de3e0f-584b9745cd-mljwh\" (UID: \"ac1fdc03-4f97-45a8-83d7-83a881b9e483\") " pod="openshift-ingress/payload-processing-e2e-derive-73de3e0f-584b9745cd-mljwh" Jul 29 04:35:20.196534 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:20.196418 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnhp9\" (UniqueName: \"kubernetes.io/projected/ac1fdc03-4f97-45a8-83d7-83a881b9e483-kube-api-access-wnhp9\") pod \"payload-processing-e2e-derive-73de3e0f-584b9745cd-mljwh\" (UID: \"ac1fdc03-4f97-45a8-83d7-83a881b9e483\") " pod="openshift-ingress/payload-processing-e2e-derive-73de3e0f-584b9745cd-mljwh" Jul 29 04:35:20.196534 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:20.196442 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/347ecb1d-a89c-49c6-a7b8-70baf108c503-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-73de3e0f-544dd76cf6-hjzm7\" (UID: \"347ecb1d-a89c-49c6-a7b8-70baf108c503\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-73de3e0f-544dd76cf6-hjzm7" Jul 29 04:35:20.297344 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:20.297253 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac1fdc03-4f97-45a8-83d7-83a881b9e483-plugins-config-volume\") pod \"payload-processing-e2e-derive-73de3e0f-584b9745cd-mljwh\" (UID: \"ac1fdc03-4f97-45a8-83d7-83a881b9e483\") " pod="openshift-ingress/payload-processing-e2e-derive-73de3e0f-584b9745cd-mljwh" Jul 29 04:35:20.297344 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:20.297307 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnhp9\" (UniqueName: \"kubernetes.io/projected/ac1fdc03-4f97-45a8-83d7-83a881b9e483-kube-api-access-wnhp9\") pod \"payload-processing-e2e-derive-73de3e0f-584b9745cd-mljwh\" (UID: \"ac1fdc03-4f97-45a8-83d7-83a881b9e483\") " pod="openshift-ingress/payload-processing-e2e-derive-73de3e0f-584b9745cd-mljwh" Jul 29 04:35:20.297544 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:20.297433 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/347ecb1d-a89c-49c6-a7b8-70baf108c503-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-73de3e0f-544dd76cf6-hjzm7\" (UID: \"347ecb1d-a89c-49c6-a7b8-70baf108c503\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-73de3e0f-544dd76cf6-hjzm7" Jul 29 04:35:20.297544 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:20.297519 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plvnl\" (UniqueName: \"kubernetes.io/projected/347ecb1d-a89c-49c6-a7b8-70baf108c503-kube-api-access-plvnl\") pod \"payload-pre-processing-e2e-derive-73de3e0f-544dd76cf6-hjzm7\" (UID: \"347ecb1d-a89c-49c6-a7b8-70baf108c503\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-73de3e0f-544dd76cf6-hjzm7" Jul 29 04:35:20.298037 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:20.298019 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac1fdc03-4f97-45a8-83d7-83a881b9e483-plugins-config-volume\") pod \"payload-processing-e2e-derive-73de3e0f-584b9745cd-mljwh\" (UID: \"ac1fdc03-4f97-45a8-83d7-83a881b9e483\") " pod="openshift-ingress/payload-processing-e2e-derive-73de3e0f-584b9745cd-mljwh" Jul 29 04:35:20.298144 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:20.298118 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/347ecb1d-a89c-49c6-a7b8-70baf108c503-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-73de3e0f-544dd76cf6-hjzm7\" (UID: \"347ecb1d-a89c-49c6-a7b8-70baf108c503\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-73de3e0f-544dd76cf6-hjzm7" Jul 29 04:35:20.304811 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:20.304788 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnhp9\" (UniqueName: \"kubernetes.io/projected/ac1fdc03-4f97-45a8-83d7-83a881b9e483-kube-api-access-wnhp9\") pod \"payload-processing-e2e-derive-73de3e0f-584b9745cd-mljwh\" (UID: \"ac1fdc03-4f97-45a8-83d7-83a881b9e483\") " pod="openshift-ingress/payload-processing-e2e-derive-73de3e0f-584b9745cd-mljwh" Jul 29 04:35:20.304947 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:20.304790 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plvnl\" (UniqueName: \"kubernetes.io/projected/347ecb1d-a89c-49c6-a7b8-70baf108c503-kube-api-access-plvnl\") pod \"payload-pre-processing-e2e-derive-73de3e0f-544dd76cf6-hjzm7\" (UID: \"347ecb1d-a89c-49c6-a7b8-70baf108c503\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-73de3e0f-544dd76cf6-hjzm7" Jul 29 04:35:20.342810 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:20.342779 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-73de3e0f-544dd76cf6-hjzm7" Jul 29 04:35:20.379423 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:20.379388 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-73de3e0f-584b9745cd-mljwh" Jul 29 04:35:20.493119 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:20.493083 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-73de3e0f-544dd76cf6-hjzm7"] Jul 29 04:35:20.496507 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:35:20.496469 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod347ecb1d_a89c_49c6_a7b8_70baf108c503.slice/crio-f84ecde80faaaaa3445d762f7772e0b03bdd7389e6697285bd03dc3a010a9629 WatchSource:0}: Error finding container f84ecde80faaaaa3445d762f7772e0b03bdd7389e6697285bd03dc3a010a9629: Status 404 returned error can't find the container with id f84ecde80faaaaa3445d762f7772e0b03bdd7389e6697285bd03dc3a010a9629 Jul 29 04:35:20.536161 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:20.536085 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-73de3e0f-584b9745cd-mljwh"] Jul 29 04:35:20.538702 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:35:20.538660 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac1fdc03_4f97_45a8_83d7_83a881b9e483.slice/crio-25a6029c121fa81a60c94126eff0e1d76a0f11fa630816f4dc482809daa1e6a8 WatchSource:0}: Error finding container 25a6029c121fa81a60c94126eff0e1d76a0f11fa630816f4dc482809daa1e6a8: Status 404 returned error can't find the container with id 25a6029c121fa81a60c94126eff0e1d76a0f11fa630816f4dc482809daa1e6a8 Jul 29 04:35:21.454271 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:21.454238 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-73de3e0f-584b9745cd-mljwh" event={"ID":"ac1fdc03-4f97-45a8-83d7-83a881b9e483","Type":"ContainerStarted","Data":"a661e438db029418becd9f2a96042e7de4e6f8bc68ad7b8a087ac1ab8bb8ec34"} Jul 29 04:35:21.454738 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:21.454299 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-73de3e0f-584b9745cd-mljwh" event={"ID":"ac1fdc03-4f97-45a8-83d7-83a881b9e483","Type":"ContainerStarted","Data":"25a6029c121fa81a60c94126eff0e1d76a0f11fa630816f4dc482809daa1e6a8"} Jul 29 04:35:21.454738 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:21.454410 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-73de3e0f-584b9745cd-mljwh" Jul 29 04:35:21.455827 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:21.455805 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-73de3e0f-544dd76cf6-hjzm7" event={"ID":"347ecb1d-a89c-49c6-a7b8-70baf108c503","Type":"ContainerStarted","Data":"5b7ac18039c48ed8ec04ce9b7bd87382e7c8708555903b1f3ada329b460718cd"} Jul 29 04:35:21.455954 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:21.455846 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-73de3e0f-544dd76cf6-hjzm7" event={"ID":"347ecb1d-a89c-49c6-a7b8-70baf108c503","Type":"ContainerStarted","Data":"f84ecde80faaaaa3445d762f7772e0b03bdd7389e6697285bd03dc3a010a9629"} Jul 29 04:35:21.455954 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:21.455872 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-73de3e0f-544dd76cf6-hjzm7" Jul 29 04:35:21.469836 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:21.469782 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-73de3e0f-584b9745cd-mljwh" podStartSLOduration=1.469759143 podStartE2EDuration="1.469759143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:35:21.469169236 +0000 UTC m=+1902.328543984" watchObservedRunningTime="2026-07-29 04:35:21.469759143 +0000 UTC m=+1902.329133871" Jul 29 04:35:21.483513 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:21.483467 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-73de3e0f-544dd76cf6-hjzm7" podStartSLOduration=1.483450749 podStartE2EDuration="1.483450749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:35:21.482942248 +0000 UTC m=+1902.342316971" watchObservedRunningTime="2026-07-29 04:35:21.483450749 +0000 UTC m=+1902.342825501" Jul 29 04:35:32.462840 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:32.462810 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-73de3e0f-584b9745cd-mljwh" Jul 29 04:35:32.463320 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:32.462876 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-73de3e0f-544dd76cf6-hjzm7" Jul 29 04:35:50.004556 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.004521 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-73de3e0f-584b9745cd-mljwh"] Jul 29 04:35:50.005084 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.004728 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-73de3e0f-584b9745cd-mljwh" podUID="ac1fdc03-4f97-45a8-83d7-83a881b9e483" containerName="payload-processing" containerID="cri-o://a661e438db029418becd9f2a96042e7de4e6f8bc68ad7b8a087ac1ab8bb8ec34" gracePeriod=30 Jul 29 04:35:50.014444 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.014414 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-73de3e0f-544dd76cf6-hjzm7"] Jul 29 04:35:50.014668 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.014640 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-73de3e0f-544dd76cf6-hjzm7" podUID="347ecb1d-a89c-49c6-a7b8-70baf108c503" containerName="payload-pre-processing" containerID="cri-o://5b7ac18039c48ed8ec04ce9b7bd87382e7c8708555903b1f3ada329b460718cd" gracePeriod=30 Jul 29 04:35:50.304919 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.304882 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-73de3e0f-584b9745cd-mljwh" Jul 29 04:35:50.307788 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.307766 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-73de3e0f-544dd76cf6-hjzm7" Jul 29 04:35:50.365036 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.365000 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plvnl\" (UniqueName: \"kubernetes.io/projected/347ecb1d-a89c-49c6-a7b8-70baf108c503-kube-api-access-plvnl\") pod \"347ecb1d-a89c-49c6-a7b8-70baf108c503\" (UID: \"347ecb1d-a89c-49c6-a7b8-70baf108c503\") " Jul 29 04:35:50.365256 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.365061 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/347ecb1d-a89c-49c6-a7b8-70baf108c503-plugins-config-volume\") pod \"347ecb1d-a89c-49c6-a7b8-70baf108c503\" (UID: \"347ecb1d-a89c-49c6-a7b8-70baf108c503\") " Jul 29 04:35:50.365256 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.365106 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnhp9\" (UniqueName: \"kubernetes.io/projected/ac1fdc03-4f97-45a8-83d7-83a881b9e483-kube-api-access-wnhp9\") pod \"ac1fdc03-4f97-45a8-83d7-83a881b9e483\" (UID: \"ac1fdc03-4f97-45a8-83d7-83a881b9e483\") " Jul 29 04:35:50.365256 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.365134 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac1fdc03-4f97-45a8-83d7-83a881b9e483-plugins-config-volume\") pod \"ac1fdc03-4f97-45a8-83d7-83a881b9e483\" (UID: \"ac1fdc03-4f97-45a8-83d7-83a881b9e483\") " Jul 29 04:35:50.365528 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.365501 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac1fdc03-4f97-45a8-83d7-83a881b9e483-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ac1fdc03-4f97-45a8-83d7-83a881b9e483" (UID: "ac1fdc03-4f97-45a8-83d7-83a881b9e483"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 04:35:50.365596 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.365506 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/347ecb1d-a89c-49c6-a7b8-70baf108c503-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "347ecb1d-a89c-49c6-a7b8-70baf108c503" (UID: "347ecb1d-a89c-49c6-a7b8-70baf108c503"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 04:35:50.367135 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.367116 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac1fdc03-4f97-45a8-83d7-83a881b9e483-kube-api-access-wnhp9" (OuterVolumeSpecName: "kube-api-access-wnhp9") pod "ac1fdc03-4f97-45a8-83d7-83a881b9e483" (UID: "ac1fdc03-4f97-45a8-83d7-83a881b9e483"). InnerVolumeSpecName "kube-api-access-wnhp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:35:50.367198 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.367161 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/347ecb1d-a89c-49c6-a7b8-70baf108c503-kube-api-access-plvnl" (OuterVolumeSpecName: "kube-api-access-plvnl") pod "347ecb1d-a89c-49c6-a7b8-70baf108c503" (UID: "347ecb1d-a89c-49c6-a7b8-70baf108c503"). InnerVolumeSpecName "kube-api-access-plvnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:35:50.466257 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.466191 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plvnl\" (UniqueName: \"kubernetes.io/projected/347ecb1d-a89c-49c6-a7b8-70baf108c503-kube-api-access-plvnl\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:35:50.466257 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.466250 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/347ecb1d-a89c-49c6-a7b8-70baf108c503-plugins-config-volume\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:35:50.466257 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.466260 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnhp9\" (UniqueName: \"kubernetes.io/projected/ac1fdc03-4f97-45a8-83d7-83a881b9e483-kube-api-access-wnhp9\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:35:50.466257 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.466269 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac1fdc03-4f97-45a8-83d7-83a881b9e483-plugins-config-volume\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:35:50.597077 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.596983 2579 generic.go:356] "Generic (PLEG): container finished" podID="ac1fdc03-4f97-45a8-83d7-83a881b9e483" containerID="a661e438db029418becd9f2a96042e7de4e6f8bc68ad7b8a087ac1ab8bb8ec34" exitCode=0 Jul 29 04:35:50.597077 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.597028 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-73de3e0f-584b9745cd-mljwh" Jul 29 04:35:50.597317 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.597067 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-73de3e0f-584b9745cd-mljwh" event={"ID":"ac1fdc03-4f97-45a8-83d7-83a881b9e483","Type":"ContainerDied","Data":"a661e438db029418becd9f2a96042e7de4e6f8bc68ad7b8a087ac1ab8bb8ec34"} Jul 29 04:35:50.597317 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.597125 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-73de3e0f-584b9745cd-mljwh" event={"ID":"ac1fdc03-4f97-45a8-83d7-83a881b9e483","Type":"ContainerDied","Data":"25a6029c121fa81a60c94126eff0e1d76a0f11fa630816f4dc482809daa1e6a8"} Jul 29 04:35:50.597317 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.597156 2579 scope.go:117] "RemoveContainer" containerID="a661e438db029418becd9f2a96042e7de4e6f8bc68ad7b8a087ac1ab8bb8ec34" Jul 29 04:35:50.598587 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.598563 2579 generic.go:356] "Generic (PLEG): container finished" podID="347ecb1d-a89c-49c6-a7b8-70baf108c503" containerID="5b7ac18039c48ed8ec04ce9b7bd87382e7c8708555903b1f3ada329b460718cd" exitCode=0 Jul 29 04:35:50.598703 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.598604 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-73de3e0f-544dd76cf6-hjzm7" Jul 29 04:35:50.598703 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.598636 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-73de3e0f-544dd76cf6-hjzm7" event={"ID":"347ecb1d-a89c-49c6-a7b8-70baf108c503","Type":"ContainerDied","Data":"5b7ac18039c48ed8ec04ce9b7bd87382e7c8708555903b1f3ada329b460718cd"} Jul 29 04:35:50.598703 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.598689 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-73de3e0f-544dd76cf6-hjzm7" event={"ID":"347ecb1d-a89c-49c6-a7b8-70baf108c503","Type":"ContainerDied","Data":"f84ecde80faaaaa3445d762f7772e0b03bdd7389e6697285bd03dc3a010a9629"} Jul 29 04:35:50.607594 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.607553 2579 scope.go:117] "RemoveContainer" containerID="a661e438db029418becd9f2a96042e7de4e6f8bc68ad7b8a087ac1ab8bb8ec34" Jul 29 04:35:50.607908 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:35:50.607866 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a661e438db029418becd9f2a96042e7de4e6f8bc68ad7b8a087ac1ab8bb8ec34\": container with ID starting with a661e438db029418becd9f2a96042e7de4e6f8bc68ad7b8a087ac1ab8bb8ec34 not found: ID does not exist" containerID="a661e438db029418becd9f2a96042e7de4e6f8bc68ad7b8a087ac1ab8bb8ec34" Jul 29 04:35:50.607908 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.607895 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a661e438db029418becd9f2a96042e7de4e6f8bc68ad7b8a087ac1ab8bb8ec34"} err="failed to get container status \"a661e438db029418becd9f2a96042e7de4e6f8bc68ad7b8a087ac1ab8bb8ec34\": rpc error: code = NotFound desc = could not find container \"a661e438db029418becd9f2a96042e7de4e6f8bc68ad7b8a087ac1ab8bb8ec34\": container with ID starting with a661e438db029418becd9f2a96042e7de4e6f8bc68ad7b8a087ac1ab8bb8ec34 not found: ID does not exist" Jul 29 04:35:50.608024 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.607913 2579 scope.go:117] "RemoveContainer" containerID="5b7ac18039c48ed8ec04ce9b7bd87382e7c8708555903b1f3ada329b460718cd" Jul 29 04:35:50.616113 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.616093 2579 scope.go:117] "RemoveContainer" containerID="5b7ac18039c48ed8ec04ce9b7bd87382e7c8708555903b1f3ada329b460718cd" Jul 29 04:35:50.616436 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:35:50.616415 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b7ac18039c48ed8ec04ce9b7bd87382e7c8708555903b1f3ada329b460718cd\": container with ID starting with 5b7ac18039c48ed8ec04ce9b7bd87382e7c8708555903b1f3ada329b460718cd not found: ID does not exist" containerID="5b7ac18039c48ed8ec04ce9b7bd87382e7c8708555903b1f3ada329b460718cd" Jul 29 04:35:50.616522 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.616444 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b7ac18039c48ed8ec04ce9b7bd87382e7c8708555903b1f3ada329b460718cd"} err="failed to get container status \"5b7ac18039c48ed8ec04ce9b7bd87382e7c8708555903b1f3ada329b460718cd\": rpc error: code = NotFound desc = could not find container \"5b7ac18039c48ed8ec04ce9b7bd87382e7c8708555903b1f3ada329b460718cd\": container with ID starting with 5b7ac18039c48ed8ec04ce9b7bd87382e7c8708555903b1f3ada329b460718cd not found: ID does not exist" Jul 29 04:35:50.623048 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.623021 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-73de3e0f-544dd76cf6-hjzm7"] Jul 29 04:35:50.625922 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.625895 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-73de3e0f-544dd76cf6-hjzm7"] Jul 29 04:35:50.634453 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.634427 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-73de3e0f-584b9745cd-mljwh"] Jul 29 04:35:50.638339 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:50.638315 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-73de3e0f-584b9745cd-mljwh"] Jul 29 04:35:51.591838 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:51.591807 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="347ecb1d-a89c-49c6-a7b8-70baf108c503" path="/var/lib/kubelet/pods/347ecb1d-a89c-49c6-a7b8-70baf108c503/volumes" Jul 29 04:35:51.592343 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:35:51.592161 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac1fdc03-4f97-45a8-83d7-83a881b9e483" path="/var/lib/kubelet/pods/ac1fdc03-4f97-45a8-83d7-83a881b9e483/volumes" Jul 29 04:39:55.742546 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:39:55.742509 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ebe3d1f6-954865dbd-vjst7"] Jul 29 04:39:55.744514 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:39:55.742889 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="347ecb1d-a89c-49c6-a7b8-70baf108c503" containerName="payload-pre-processing" Jul 29 04:39:55.744514 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:39:55.742902 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="347ecb1d-a89c-49c6-a7b8-70baf108c503" containerName="payload-pre-processing" Jul 29 04:39:55.744514 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:39:55.742918 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac1fdc03-4f97-45a8-83d7-83a881b9e483" containerName="payload-processing" Jul 29 04:39:55.744514 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:39:55.742925 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac1fdc03-4f97-45a8-83d7-83a881b9e483" containerName="payload-processing" Jul 29 04:39:55.744514 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:39:55.742989 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac1fdc03-4f97-45a8-83d7-83a881b9e483" containerName="payload-processing" Jul 29 04:39:55.744514 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:39:55.742999 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="347ecb1d-a89c-49c6-a7b8-70baf108c503" containerName="payload-pre-processing" Jul 29 04:39:55.746035 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:39:55.746020 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ebe3d1f6-954865dbd-vjst7" Jul 29 04:39:55.748518 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:39:55.748491 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ebe3d1f6\"" Jul 29 04:39:55.754221 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:39:55.749224 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ebe3d1f6-dockercfg-t8v8t\"" Jul 29 04:39:55.757710 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:39:55.757471 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ebe3d1f6-954865dbd-vjst7"] Jul 29 04:39:55.921319 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:39:55.921279 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5d88fb5-915f-4592-8985-52deca022469-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ebe3d1f6-954865dbd-vjst7\" (UID: \"a5d88fb5-915f-4592-8985-52deca022469\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ebe3d1f6-954865dbd-vjst7" Jul 29 04:39:55.921493 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:39:55.921326 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crwbc\" (UniqueName: \"kubernetes.io/projected/a5d88fb5-915f-4592-8985-52deca022469-kube-api-access-crwbc\") pod \"payload-pre-processing-e2e-mt-ebe3d1f6-954865dbd-vjst7\" (UID: \"a5d88fb5-915f-4592-8985-52deca022469\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ebe3d1f6-954865dbd-vjst7" Jul 29 04:39:56.022923 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:39:56.022836 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5d88fb5-915f-4592-8985-52deca022469-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ebe3d1f6-954865dbd-vjst7\" (UID: \"a5d88fb5-915f-4592-8985-52deca022469\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ebe3d1f6-954865dbd-vjst7" Jul 29 04:39:56.022923 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:39:56.022871 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crwbc\" (UniqueName: \"kubernetes.io/projected/a5d88fb5-915f-4592-8985-52deca022469-kube-api-access-crwbc\") pod \"payload-pre-processing-e2e-mt-ebe3d1f6-954865dbd-vjst7\" (UID: \"a5d88fb5-915f-4592-8985-52deca022469\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ebe3d1f6-954865dbd-vjst7" Jul 29 04:39:56.023596 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:39:56.023575 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5d88fb5-915f-4592-8985-52deca022469-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ebe3d1f6-954865dbd-vjst7\" (UID: \"a5d88fb5-915f-4592-8985-52deca022469\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ebe3d1f6-954865dbd-vjst7" Jul 29 04:39:56.035966 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:39:56.035933 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crwbc\" (UniqueName: \"kubernetes.io/projected/a5d88fb5-915f-4592-8985-52deca022469-kube-api-access-crwbc\") pod \"payload-pre-processing-e2e-mt-ebe3d1f6-954865dbd-vjst7\" (UID: \"a5d88fb5-915f-4592-8985-52deca022469\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ebe3d1f6-954865dbd-vjst7" Jul 29 04:39:56.062556 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:39:56.062529 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ebe3d1f6-954865dbd-vjst7" Jul 29 04:39:56.408936 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:39:56.408909 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ebe3d1f6-954865dbd-vjst7"] Jul 29 04:39:56.410813 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:39:56.410784 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d88fb5_915f_4592_8985_52deca022469.slice/crio-14cbe568b9cb07dc7b500b93bc330bd37af2969b206cc730ed438418b12432c9 WatchSource:0}: Error finding container 14cbe568b9cb07dc7b500b93bc330bd37af2969b206cc730ed438418b12432c9: Status 404 returned error can't find the container with id 14cbe568b9cb07dc7b500b93bc330bd37af2969b206cc730ed438418b12432c9 Jul 29 04:39:56.814165 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:39:56.814112 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ebe3d1f6-954865dbd-vjst7" event={"ID":"a5d88fb5-915f-4592-8985-52deca022469","Type":"ContainerStarted","Data":"285d454a11990b45e89aee4757c20c15737bf11969f4e94b67383240c49ab020"} Jul 29 04:39:56.814610 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:39:56.814190 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ebe3d1f6-954865dbd-vjst7" Jul 29 04:39:56.814610 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:39:56.814245 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ebe3d1f6-954865dbd-vjst7" event={"ID":"a5d88fb5-915f-4592-8985-52deca022469","Type":"ContainerStarted","Data":"14cbe568b9cb07dc7b500b93bc330bd37af2969b206cc730ed438418b12432c9"} Jul 29 04:39:56.835075 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:39:56.835010 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ebe3d1f6-954865dbd-vjst7" podStartSLOduration=1.834990645 podStartE2EDuration="1.834990645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:39:56.832224354 +0000 UTC m=+2177.691599081" watchObservedRunningTime="2026-07-29 04:39:56.834990645 +0000 UTC m=+2177.694365371" Jul 29 04:40:07.820718 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:40:07.820690 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ebe3d1f6-954865dbd-vjst7" Jul 29 04:40:37.472554 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:40:37.472517 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ebe3d1f6-954865dbd-vjst7"] Jul 29 04:40:37.473129 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:40:37.472797 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ebe3d1f6-954865dbd-vjst7" podUID="a5d88fb5-915f-4592-8985-52deca022469" containerName="payload-pre-processing" containerID="cri-o://285d454a11990b45e89aee4757c20c15737bf11969f4e94b67383240c49ab020" gracePeriod=30 Jul 29 04:40:37.735447 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:40:37.735383 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ebe3d1f6-954865dbd-vjst7" Jul 29 04:40:37.890448 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:40:37.890411 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5d88fb5-915f-4592-8985-52deca022469-plugins-config-volume\") pod \"a5d88fb5-915f-4592-8985-52deca022469\" (UID: \"a5d88fb5-915f-4592-8985-52deca022469\") " Jul 29 04:40:37.890647 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:40:37.890459 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crwbc\" (UniqueName: \"kubernetes.io/projected/a5d88fb5-915f-4592-8985-52deca022469-kube-api-access-crwbc\") pod \"a5d88fb5-915f-4592-8985-52deca022469\" (UID: \"a5d88fb5-915f-4592-8985-52deca022469\") " Jul 29 04:40:37.890848 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:40:37.890820 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5d88fb5-915f-4592-8985-52deca022469-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a5d88fb5-915f-4592-8985-52deca022469" (UID: "a5d88fb5-915f-4592-8985-52deca022469"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 04:40:37.892482 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:40:37.892456 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5d88fb5-915f-4592-8985-52deca022469-kube-api-access-crwbc" (OuterVolumeSpecName: "kube-api-access-crwbc") pod "a5d88fb5-915f-4592-8985-52deca022469" (UID: "a5d88fb5-915f-4592-8985-52deca022469"). InnerVolumeSpecName "kube-api-access-crwbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:40:37.991774 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:40:37.991690 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5d88fb5-915f-4592-8985-52deca022469-plugins-config-volume\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:40:37.991774 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:40:37.991719 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crwbc\" (UniqueName: \"kubernetes.io/projected/a5d88fb5-915f-4592-8985-52deca022469-kube-api-access-crwbc\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:40:38.015779 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:40:38.015748 2579 generic.go:356] "Generic (PLEG): container finished" podID="a5d88fb5-915f-4592-8985-52deca022469" containerID="285d454a11990b45e89aee4757c20c15737bf11969f4e94b67383240c49ab020" exitCode=0 Jul 29 04:40:38.015953 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:40:38.015791 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ebe3d1f6-954865dbd-vjst7" Jul 29 04:40:38.015953 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:40:38.015843 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ebe3d1f6-954865dbd-vjst7" event={"ID":"a5d88fb5-915f-4592-8985-52deca022469","Type":"ContainerDied","Data":"285d454a11990b45e89aee4757c20c15737bf11969f4e94b67383240c49ab020"} Jul 29 04:40:38.015953 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:40:38.015900 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ebe3d1f6-954865dbd-vjst7" event={"ID":"a5d88fb5-915f-4592-8985-52deca022469","Type":"ContainerDied","Data":"14cbe568b9cb07dc7b500b93bc330bd37af2969b206cc730ed438418b12432c9"} Jul 29 04:40:38.015953 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:40:38.015924 2579 scope.go:117] "RemoveContainer" containerID="285d454a11990b45e89aee4757c20c15737bf11969f4e94b67383240c49ab020" Jul 29 04:40:38.027699 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:40:38.027665 2579 scope.go:117] "RemoveContainer" containerID="285d454a11990b45e89aee4757c20c15737bf11969f4e94b67383240c49ab020" Jul 29 04:40:38.028143 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:40:38.028117 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"285d454a11990b45e89aee4757c20c15737bf11969f4e94b67383240c49ab020\": container with ID starting with 285d454a11990b45e89aee4757c20c15737bf11969f4e94b67383240c49ab020 not found: ID does not exist" containerID="285d454a11990b45e89aee4757c20c15737bf11969f4e94b67383240c49ab020" Jul 29 04:40:38.028258 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:40:38.028156 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"285d454a11990b45e89aee4757c20c15737bf11969f4e94b67383240c49ab020"} err="failed to get container status \"285d454a11990b45e89aee4757c20c15737bf11969f4e94b67383240c49ab020\": rpc error: code = NotFound desc = could not find container \"285d454a11990b45e89aee4757c20c15737bf11969f4e94b67383240c49ab020\": container with ID starting with 285d454a11990b45e89aee4757c20c15737bf11969f4e94b67383240c49ab020 not found: ID does not exist" Jul 29 04:40:38.046727 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:40:38.046700 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ebe3d1f6-954865dbd-vjst7"] Jul 29 04:40:38.051245 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:40:38.051224 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ebe3d1f6-954865dbd-vjst7"] Jul 29 04:40:39.591971 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:40:39.591939 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5d88fb5-915f-4592-8985-52deca022469" path="/var/lib/kubelet/pods/a5d88fb5-915f-4592-8985-52deca022469/volumes" Jul 29 04:40:39.765080 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:40:39.765052 2579 scope.go:117] "RemoveContainer" containerID="ad81846d95544a559cc049c23ec85f5cd0cfad3379369fc653763bbafb171aac" Jul 29 04:41:35.239447 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:41:35.239408 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szn9p"] Jul 29 04:41:35.240113 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:41:35.239663 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szn9p" podUID="9a2a3d4c-6605-4e87-bb39-ec1402d0d894" containerName="manager" containerID="cri-o://fdf5d264d9dc4bf26a6e17d41092f7694ffc80efa1fb44bba7f4b42f476f66aa" gracePeriod=10 Jul 29 04:41:36.201626 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:41:36.201599 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szn9p" Jul 29 04:41:36.288069 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:41:36.288035 2579 generic.go:356] "Generic (PLEG): container finished" podID="9a2a3d4c-6605-4e87-bb39-ec1402d0d894" containerID="fdf5d264d9dc4bf26a6e17d41092f7694ffc80efa1fb44bba7f4b42f476f66aa" exitCode=0 Jul 29 04:41:36.288617 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:41:36.288078 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szn9p" Jul 29 04:41:36.288617 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:41:36.288091 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szn9p" event={"ID":"9a2a3d4c-6605-4e87-bb39-ec1402d0d894","Type":"ContainerDied","Data":"fdf5d264d9dc4bf26a6e17d41092f7694ffc80efa1fb44bba7f4b42f476f66aa"} Jul 29 04:41:36.288617 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:41:36.288144 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szn9p" event={"ID":"9a2a3d4c-6605-4e87-bb39-ec1402d0d894","Type":"ContainerDied","Data":"1b46ff09267b09080650e49ec01d6847d5f5200676bf485d56dc90c78f0d7545"} Jul 29 04:41:36.288617 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:41:36.288173 2579 scope.go:117] "RemoveContainer" containerID="fdf5d264d9dc4bf26a6e17d41092f7694ffc80efa1fb44bba7f4b42f476f66aa" Jul 29 04:41:36.298122 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:41:36.298100 2579 scope.go:117] "RemoveContainer" containerID="fdf5d264d9dc4bf26a6e17d41092f7694ffc80efa1fb44bba7f4b42f476f66aa" Jul 29 04:41:36.298446 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:41:36.298427 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdf5d264d9dc4bf26a6e17d41092f7694ffc80efa1fb44bba7f4b42f476f66aa\": container with ID starting with fdf5d264d9dc4bf26a6e17d41092f7694ffc80efa1fb44bba7f4b42f476f66aa not found: ID does not exist" containerID="fdf5d264d9dc4bf26a6e17d41092f7694ffc80efa1fb44bba7f4b42f476f66aa" Jul 29 04:41:36.298509 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:41:36.298454 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdf5d264d9dc4bf26a6e17d41092f7694ffc80efa1fb44bba7f4b42f476f66aa"} err="failed to get container status \"fdf5d264d9dc4bf26a6e17d41092f7694ffc80efa1fb44bba7f4b42f476f66aa\": rpc error: code = NotFound desc = could not find container \"fdf5d264d9dc4bf26a6e17d41092f7694ffc80efa1fb44bba7f4b42f476f66aa\": container with ID starting with fdf5d264d9dc4bf26a6e17d41092f7694ffc80efa1fb44bba7f4b42f476f66aa not found: ID does not exist" Jul 29 04:41:36.319378 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:41:36.319317 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9a2a3d4c-6605-4e87-bb39-ec1402d0d894-extensions-socket-volume\") pod \"9a2a3d4c-6605-4e87-bb39-ec1402d0d894\" (UID: \"9a2a3d4c-6605-4e87-bb39-ec1402d0d894\") " Jul 29 04:41:36.319378 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:41:36.319365 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2smh9\" (UniqueName: \"kubernetes.io/projected/9a2a3d4c-6605-4e87-bb39-ec1402d0d894-kube-api-access-2smh9\") pod \"9a2a3d4c-6605-4e87-bb39-ec1402d0d894\" (UID: \"9a2a3d4c-6605-4e87-bb39-ec1402d0d894\") " Jul 29 04:41:36.319713 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:41:36.319692 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a2a3d4c-6605-4e87-bb39-ec1402d0d894-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9a2a3d4c-6605-4e87-bb39-ec1402d0d894" (UID: "9a2a3d4c-6605-4e87-bb39-ec1402d0d894"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 04:41:36.321312 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:41:36.321294 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a2a3d4c-6605-4e87-bb39-ec1402d0d894-kube-api-access-2smh9" (OuterVolumeSpecName: "kube-api-access-2smh9") pod "9a2a3d4c-6605-4e87-bb39-ec1402d0d894" (UID: "9a2a3d4c-6605-4e87-bb39-ec1402d0d894"). InnerVolumeSpecName "kube-api-access-2smh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:41:36.420590 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:41:36.420558 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9a2a3d4c-6605-4e87-bb39-ec1402d0d894-extensions-socket-volume\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:41:36.420590 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:41:36.420586 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2smh9\" (UniqueName: \"kubernetes.io/projected/9a2a3d4c-6605-4e87-bb39-ec1402d0d894-kube-api-access-2smh9\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:41:36.609862 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:41:36.609831 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szn9p"] Jul 29 04:41:36.617889 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:41:36.617860 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szn9p"] Jul 29 04:41:37.591148 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:41:37.591101 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a2a3d4c-6605-4e87-bb39-ec1402d0d894" path="/var/lib/kubelet/pods/9a2a3d4c-6605-4e87-bb39-ec1402d0d894/volumes" Jul 29 04:42:50.836714 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:42:50.836684 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-774gm"] Jul 29 04:42:50.837184 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:42:50.837045 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5d88fb5-915f-4592-8985-52deca022469" containerName="payload-pre-processing" Jul 29 04:42:50.837184 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:42:50.837056 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5d88fb5-915f-4592-8985-52deca022469" containerName="payload-pre-processing" Jul 29 04:42:50.837184 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:42:50.837066 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a2a3d4c-6605-4e87-bb39-ec1402d0d894" containerName="manager" Jul 29 04:42:50.837184 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:42:50.837072 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a2a3d4c-6605-4e87-bb39-ec1402d0d894" containerName="manager" Jul 29 04:42:50.837184 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:42:50.837123 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5d88fb5-915f-4592-8985-52deca022469" containerName="payload-pre-processing" Jul 29 04:42:50.837184 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:42:50.837131 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a2a3d4c-6605-4e87-bb39-ec1402d0d894" containerName="manager" Jul 29 04:42:50.840073 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:42:50.840057 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-774gm" Jul 29 04:42:50.842888 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:42:50.842865 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-nkp54\"" Jul 29 04:42:50.852321 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:42:50.852296 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-774gm"] Jul 29 04:42:50.857685 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:42:50.857661 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj2xx\" (UniqueName: \"kubernetes.io/projected/f724cd5b-b799-4a64-8a7b-109b83601d2b-kube-api-access-pj2xx\") pod \"kuadrant-operator-controller-manager-86878c769f-774gm\" (UID: \"f724cd5b-b799-4a64-8a7b-109b83601d2b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-774gm" Jul 29 04:42:50.857790 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:42:50.857697 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f724cd5b-b799-4a64-8a7b-109b83601d2b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-774gm\" (UID: \"f724cd5b-b799-4a64-8a7b-109b83601d2b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-774gm" Jul 29 04:42:50.958844 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:42:50.958802 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj2xx\" (UniqueName: \"kubernetes.io/projected/f724cd5b-b799-4a64-8a7b-109b83601d2b-kube-api-access-pj2xx\") pod \"kuadrant-operator-controller-manager-86878c769f-774gm\" (UID: \"f724cd5b-b799-4a64-8a7b-109b83601d2b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-774gm" Jul 29 04:42:50.959054 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:42:50.958854 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f724cd5b-b799-4a64-8a7b-109b83601d2b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-774gm\" (UID: \"f724cd5b-b799-4a64-8a7b-109b83601d2b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-774gm" Jul 29 04:42:50.959315 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:42:50.959293 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f724cd5b-b799-4a64-8a7b-109b83601d2b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-774gm\" (UID: \"f724cd5b-b799-4a64-8a7b-109b83601d2b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-774gm" Jul 29 04:42:50.966363 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:42:50.966342 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj2xx\" (UniqueName: \"kubernetes.io/projected/f724cd5b-b799-4a64-8a7b-109b83601d2b-kube-api-access-pj2xx\") pod \"kuadrant-operator-controller-manager-86878c769f-774gm\" (UID: \"f724cd5b-b799-4a64-8a7b-109b83601d2b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-774gm" Jul 29 04:42:51.150433 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:42:51.150343 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-774gm" Jul 29 04:42:51.306618 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:42:51.306471 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-774gm"] Jul 29 04:42:51.309656 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:42:51.309626 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf724cd5b_b799_4a64_8a7b_109b83601d2b.slice/crio-8c86f9ee8c988112ce4040a3ade33e8ed373acb361563631ae43e4e68184cf52 WatchSource:0}: Error finding container 8c86f9ee8c988112ce4040a3ade33e8ed373acb361563631ae43e4e68184cf52: Status 404 returned error can't find the container with id 8c86f9ee8c988112ce4040a3ade33e8ed373acb361563631ae43e4e68184cf52 Jul 29 04:42:51.641733 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:42:51.641693 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-774gm" event={"ID":"f724cd5b-b799-4a64-8a7b-109b83601d2b","Type":"ContainerStarted","Data":"177be331239be4fe014af7e7417fc8aa203e6b6287890fd8244943a3415cd69b"} Jul 29 04:42:51.641928 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:42:51.641758 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-774gm" Jul 29 04:42:51.641928 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:42:51.641780 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-774gm" event={"ID":"f724cd5b-b799-4a64-8a7b-109b83601d2b","Type":"ContainerStarted","Data":"8c86f9ee8c988112ce4040a3ade33e8ed373acb361563631ae43e4e68184cf52"} Jul 29 04:42:51.662575 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:42:51.662524 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-774gm" podStartSLOduration=1.662507178 podStartE2EDuration="1.662507178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:42:51.66068383 +0000 UTC m=+2352.520058555" watchObservedRunningTime="2026-07-29 04:42:51.662507178 +0000 UTC m=+2352.521881901" Jul 29 04:43:02.649175 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:02.649144 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-774gm" Jul 29 04:43:13.393526 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:13.393487 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-994302ee-7478b885b8-fjxn2"] Jul 29 04:43:13.397308 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:13.397286 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-994302ee-7478b885b8-fjxn2" Jul 29 04:43:13.399655 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:13.399630 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-994302ee\"" Jul 29 04:43:13.399804 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:13.399702 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-994302ee-dockercfg-jjv9w\"" Jul 29 04:43:13.407718 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:13.407691 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-994302ee-7478b885b8-fjxn2"] Jul 29 04:43:13.428101 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:13.428068 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-994302ee-8669496594-mrz46"] Jul 29 04:43:13.431522 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:13.431500 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-994302ee-8669496594-mrz46" Jul 29 04:43:13.446029 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:13.445998 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-994302ee-8669496594-mrz46"] Jul 29 04:43:13.465939 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:13.465897 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jfr7\" (UniqueName: \"kubernetes.io/projected/a4b171c8-e2b2-4d45-81df-abed04234b5b-kube-api-access-6jfr7\") pod \"payload-pre-processing-e2e-mt-994302ee-7478b885b8-fjxn2\" (UID: \"a4b171c8-e2b2-4d45-81df-abed04234b5b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-994302ee-7478b885b8-fjxn2" Jul 29 04:43:13.466105 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:13.466081 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4b171c8-e2b2-4d45-81df-abed04234b5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-994302ee-7478b885b8-fjxn2\" (UID: \"a4b171c8-e2b2-4d45-81df-abed04234b5b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-994302ee-7478b885b8-fjxn2" Jul 29 04:43:13.567493 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:13.567453 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4b171c8-e2b2-4d45-81df-abed04234b5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-994302ee-7478b885b8-fjxn2\" (UID: \"a4b171c8-e2b2-4d45-81df-abed04234b5b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-994302ee-7478b885b8-fjxn2" Jul 29 04:43:13.567706 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:13.567519 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jfr7\" (UniqueName: \"kubernetes.io/projected/a4b171c8-e2b2-4d45-81df-abed04234b5b-kube-api-access-6jfr7\") pod \"payload-pre-processing-e2e-mt-994302ee-7478b885b8-fjxn2\" (UID: \"a4b171c8-e2b2-4d45-81df-abed04234b5b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-994302ee-7478b885b8-fjxn2" Jul 29 04:43:13.567706 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:13.567576 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/100a7a35-72de-4640-8273-47572d909dbd-plugins-config-volume\") pod \"payload-processing-e2e-mt-994302ee-8669496594-mrz46\" (UID: \"100a7a35-72de-4640-8273-47572d909dbd\") " pod="openshift-ingress/payload-processing-e2e-mt-994302ee-8669496594-mrz46" Jul 29 04:43:13.567706 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:13.567599 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4k67\" (UniqueName: \"kubernetes.io/projected/100a7a35-72de-4640-8273-47572d909dbd-kube-api-access-s4k67\") pod \"payload-processing-e2e-mt-994302ee-8669496594-mrz46\" (UID: \"100a7a35-72de-4640-8273-47572d909dbd\") " pod="openshift-ingress/payload-processing-e2e-mt-994302ee-8669496594-mrz46" Jul 29 04:43:13.568396 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:13.568362 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4b171c8-e2b2-4d45-81df-abed04234b5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-994302ee-7478b885b8-fjxn2\" (UID: \"a4b171c8-e2b2-4d45-81df-abed04234b5b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-994302ee-7478b885b8-fjxn2" Jul 29 04:43:13.575374 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:13.575342 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jfr7\" (UniqueName: \"kubernetes.io/projected/a4b171c8-e2b2-4d45-81df-abed04234b5b-kube-api-access-6jfr7\") pod \"payload-pre-processing-e2e-mt-994302ee-7478b885b8-fjxn2\" (UID: \"a4b171c8-e2b2-4d45-81df-abed04234b5b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-994302ee-7478b885b8-fjxn2" Jul 29 04:43:13.668341 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:13.668239 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/100a7a35-72de-4640-8273-47572d909dbd-plugins-config-volume\") pod \"payload-processing-e2e-mt-994302ee-8669496594-mrz46\" (UID: \"100a7a35-72de-4640-8273-47572d909dbd\") " pod="openshift-ingress/payload-processing-e2e-mt-994302ee-8669496594-mrz46" Jul 29 04:43:13.668341 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:13.668280 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4k67\" (UniqueName: \"kubernetes.io/projected/100a7a35-72de-4640-8273-47572d909dbd-kube-api-access-s4k67\") pod \"payload-processing-e2e-mt-994302ee-8669496594-mrz46\" (UID: \"100a7a35-72de-4640-8273-47572d909dbd\") " pod="openshift-ingress/payload-processing-e2e-mt-994302ee-8669496594-mrz46" Jul 29 04:43:13.668973 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:13.668949 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/100a7a35-72de-4640-8273-47572d909dbd-plugins-config-volume\") pod \"payload-processing-e2e-mt-994302ee-8669496594-mrz46\" (UID: \"100a7a35-72de-4640-8273-47572d909dbd\") " pod="openshift-ingress/payload-processing-e2e-mt-994302ee-8669496594-mrz46" Jul 29 04:43:13.678923 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:13.678880 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4k67\" (UniqueName: \"kubernetes.io/projected/100a7a35-72de-4640-8273-47572d909dbd-kube-api-access-s4k67\") pod \"payload-processing-e2e-mt-994302ee-8669496594-mrz46\" (UID: \"100a7a35-72de-4640-8273-47572d909dbd\") " pod="openshift-ingress/payload-processing-e2e-mt-994302ee-8669496594-mrz46" Jul 29 04:43:13.709776 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:13.709736 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-994302ee-7478b885b8-fjxn2" Jul 29 04:43:13.743516 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:13.743486 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-994302ee-8669496594-mrz46" Jul 29 04:43:13.888182 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:13.888146 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-994302ee-7478b885b8-fjxn2"] Jul 29 04:43:13.892744 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:43:13.892699 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b171c8_e2b2_4d45_81df_abed04234b5b.slice/crio-24d15c07c4ff2021923eb218d4feef86ace16fe483912ff009e3f69762328809 WatchSource:0}: Error finding container 24d15c07c4ff2021923eb218d4feef86ace16fe483912ff009e3f69762328809: Status 404 returned error can't find the container with id 24d15c07c4ff2021923eb218d4feef86ace16fe483912ff009e3f69762328809 Jul 29 04:43:14.121012 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:14.120971 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-994302ee-8669496594-mrz46"] Jul 29 04:43:14.121458 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:43:14.121430 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100a7a35_72de_4640_8273_47572d909dbd.slice/crio-b8966521e148def14511d003e041b0e06dfbec23f2015f34ea384d466e209d42 WatchSource:0}: Error finding container b8966521e148def14511d003e041b0e06dfbec23f2015f34ea384d466e209d42: Status 404 returned error can't find the container with id b8966521e148def14511d003e041b0e06dfbec23f2015f34ea384d466e209d42 Jul 29 04:43:14.757533 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:14.757492 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-994302ee-8669496594-mrz46" event={"ID":"100a7a35-72de-4640-8273-47572d909dbd","Type":"ContainerStarted","Data":"73a2686a03a5232cf37db6d16f9fc85687c65800ee78ee64ab1f20a9dd0000d1"} Jul 29 04:43:14.758015 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:14.757560 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-994302ee-8669496594-mrz46" event={"ID":"100a7a35-72de-4640-8273-47572d909dbd","Type":"ContainerStarted","Data":"b8966521e148def14511d003e041b0e06dfbec23f2015f34ea384d466e209d42"} Jul 29 04:43:14.758015 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:14.757701 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-994302ee-8669496594-mrz46" Jul 29 04:43:14.759269 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:14.759244 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-994302ee-7478b885b8-fjxn2" event={"ID":"a4b171c8-e2b2-4d45-81df-abed04234b5b","Type":"ContainerStarted","Data":"3b2a0c70fdf5e5df6d51d9f02d4e5f5cdd690b7bf22d9fd1b41e475279e43f76"} Jul 29 04:43:14.759385 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:14.759293 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-994302ee-7478b885b8-fjxn2" event={"ID":"a4b171c8-e2b2-4d45-81df-abed04234b5b","Type":"ContainerStarted","Data":"24d15c07c4ff2021923eb218d4feef86ace16fe483912ff009e3f69762328809"} Jul 29 04:43:14.759385 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:14.759360 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-994302ee-7478b885b8-fjxn2" Jul 29 04:43:14.776457 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:14.776411 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-994302ee-8669496594-mrz46" podStartSLOduration=1.77639638 podStartE2EDuration="1.77639638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:43:14.774132945 +0000 UTC m=+2375.633507682" watchObservedRunningTime="2026-07-29 04:43:14.77639638 +0000 UTC m=+2375.635771103" Jul 29 04:43:14.791418 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:14.791370 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-994302ee-7478b885b8-fjxn2" podStartSLOduration=1.791354163 podStartE2EDuration="1.791354163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:43:14.789517731 +0000 UTC m=+2375.648892455" watchObservedRunningTime="2026-07-29 04:43:14.791354163 +0000 UTC m=+2375.650728898" Jul 29 04:43:25.766755 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:25.766722 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-994302ee-8669496594-mrz46" Jul 29 04:43:25.767391 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:43:25.766889 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-994302ee-7478b885b8-fjxn2" Jul 29 04:44:37.260324 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:37.260288 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-994302ee-8669496594-mrz46"] Jul 29 04:44:37.260916 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:37.260540 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-994302ee-8669496594-mrz46" podUID="100a7a35-72de-4640-8273-47572d909dbd" containerName="payload-processing" containerID="cri-o://73a2686a03a5232cf37db6d16f9fc85687c65800ee78ee64ab1f20a9dd0000d1" gracePeriod=30 Jul 29 04:44:37.265764 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:37.265734 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-994302ee-7478b885b8-fjxn2"] Jul 29 04:44:37.266885 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:37.266830 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-994302ee-7478b885b8-fjxn2" podUID="a4b171c8-e2b2-4d45-81df-abed04234b5b" containerName="payload-pre-processing" containerID="cri-o://3b2a0c70fdf5e5df6d51d9f02d4e5f5cdd690b7bf22d9fd1b41e475279e43f76" gracePeriod=30 Jul 29 04:44:37.539263 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:37.539242 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-994302ee-8669496594-mrz46" Jul 29 04:44:37.550927 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:37.550906 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-994302ee-7478b885b8-fjxn2" Jul 29 04:44:37.605747 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:37.605722 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4b171c8-e2b2-4d45-81df-abed04234b5b-plugins-config-volume\") pod \"a4b171c8-e2b2-4d45-81df-abed04234b5b\" (UID: \"a4b171c8-e2b2-4d45-81df-abed04234b5b\") " Jul 29 04:44:37.605896 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:37.605814 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jfr7\" (UniqueName: \"kubernetes.io/projected/a4b171c8-e2b2-4d45-81df-abed04234b5b-kube-api-access-6jfr7\") pod \"a4b171c8-e2b2-4d45-81df-abed04234b5b\" (UID: \"a4b171c8-e2b2-4d45-81df-abed04234b5b\") " Jul 29 04:44:37.605896 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:37.605842 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/100a7a35-72de-4640-8273-47572d909dbd-plugins-config-volume\") pod \"100a7a35-72de-4640-8273-47572d909dbd\" (UID: \"100a7a35-72de-4640-8273-47572d909dbd\") " Jul 29 04:44:37.605896 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:37.605878 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4k67\" (UniqueName: \"kubernetes.io/projected/100a7a35-72de-4640-8273-47572d909dbd-kube-api-access-s4k67\") pod \"100a7a35-72de-4640-8273-47572d909dbd\" (UID: \"100a7a35-72de-4640-8273-47572d909dbd\") " Jul 29 04:44:37.606154 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:37.606119 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4b171c8-e2b2-4d45-81df-abed04234b5b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a4b171c8-e2b2-4d45-81df-abed04234b5b" (UID: "a4b171c8-e2b2-4d45-81df-abed04234b5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 04:44:37.606226 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:37.606195 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/100a7a35-72de-4640-8273-47572d909dbd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "100a7a35-72de-4640-8273-47572d909dbd" (UID: "100a7a35-72de-4640-8273-47572d909dbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 04:44:37.607866 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:37.607842 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b171c8-e2b2-4d45-81df-abed04234b5b-kube-api-access-6jfr7" (OuterVolumeSpecName: "kube-api-access-6jfr7") pod "a4b171c8-e2b2-4d45-81df-abed04234b5b" (UID: "a4b171c8-e2b2-4d45-81df-abed04234b5b"). InnerVolumeSpecName "kube-api-access-6jfr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:44:37.607980 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:37.607957 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/100a7a35-72de-4640-8273-47572d909dbd-kube-api-access-s4k67" (OuterVolumeSpecName: "kube-api-access-s4k67") pod "100a7a35-72de-4640-8273-47572d909dbd" (UID: "100a7a35-72de-4640-8273-47572d909dbd"). InnerVolumeSpecName "kube-api-access-s4k67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:44:37.707154 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:37.707116 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jfr7\" (UniqueName: \"kubernetes.io/projected/a4b171c8-e2b2-4d45-81df-abed04234b5b-kube-api-access-6jfr7\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:44:37.707154 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:37.707147 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/100a7a35-72de-4640-8273-47572d909dbd-plugins-config-volume\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:44:37.707154 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:37.707156 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4k67\" (UniqueName: \"kubernetes.io/projected/100a7a35-72de-4640-8273-47572d909dbd-kube-api-access-s4k67\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:44:37.707403 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:37.707165 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4b171c8-e2b2-4d45-81df-abed04234b5b-plugins-config-volume\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:44:38.157570 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:38.157534 2579 generic.go:356] "Generic (PLEG): container finished" podID="100a7a35-72de-4640-8273-47572d909dbd" containerID="73a2686a03a5232cf37db6d16f9fc85687c65800ee78ee64ab1f20a9dd0000d1" exitCode=0 Jul 29 04:44:38.157768 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:38.157586 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-994302ee-8669496594-mrz46" Jul 29 04:44:38.157768 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:38.157614 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-994302ee-8669496594-mrz46" event={"ID":"100a7a35-72de-4640-8273-47572d909dbd","Type":"ContainerDied","Data":"73a2686a03a5232cf37db6d16f9fc85687c65800ee78ee64ab1f20a9dd0000d1"} Jul 29 04:44:38.157768 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:38.157665 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-994302ee-8669496594-mrz46" event={"ID":"100a7a35-72de-4640-8273-47572d909dbd","Type":"ContainerDied","Data":"b8966521e148def14511d003e041b0e06dfbec23f2015f34ea384d466e209d42"} Jul 29 04:44:38.157768 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:38.157691 2579 scope.go:117] "RemoveContainer" containerID="73a2686a03a5232cf37db6d16f9fc85687c65800ee78ee64ab1f20a9dd0000d1" Jul 29 04:44:38.166165 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:38.166121 2579 generic.go:356] "Generic (PLEG): container finished" podID="a4b171c8-e2b2-4d45-81df-abed04234b5b" containerID="3b2a0c70fdf5e5df6d51d9f02d4e5f5cdd690b7bf22d9fd1b41e475279e43f76" exitCode=0 Jul 29 04:44:38.166325 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:38.166268 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-994302ee-7478b885b8-fjxn2" event={"ID":"a4b171c8-e2b2-4d45-81df-abed04234b5b","Type":"ContainerDied","Data":"3b2a0c70fdf5e5df6d51d9f02d4e5f5cdd690b7bf22d9fd1b41e475279e43f76"} Jul 29 04:44:38.166426 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:38.166320 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-994302ee-7478b885b8-fjxn2" Jul 29 04:44:38.166426 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:38.166333 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-994302ee-7478b885b8-fjxn2" event={"ID":"a4b171c8-e2b2-4d45-81df-abed04234b5b","Type":"ContainerDied","Data":"24d15c07c4ff2021923eb218d4feef86ace16fe483912ff009e3f69762328809"} Jul 29 04:44:38.177723 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:38.177696 2579 scope.go:117] "RemoveContainer" containerID="73a2686a03a5232cf37db6d16f9fc85687c65800ee78ee64ab1f20a9dd0000d1" Jul 29 04:44:38.178029 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:44:38.178006 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73a2686a03a5232cf37db6d16f9fc85687c65800ee78ee64ab1f20a9dd0000d1\": container with ID starting with 73a2686a03a5232cf37db6d16f9fc85687c65800ee78ee64ab1f20a9dd0000d1 not found: ID does not exist" containerID="73a2686a03a5232cf37db6d16f9fc85687c65800ee78ee64ab1f20a9dd0000d1" Jul 29 04:44:38.178088 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:38.178039 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a2686a03a5232cf37db6d16f9fc85687c65800ee78ee64ab1f20a9dd0000d1"} err="failed to get container status \"73a2686a03a5232cf37db6d16f9fc85687c65800ee78ee64ab1f20a9dd0000d1\": rpc error: code = NotFound desc = could not find container \"73a2686a03a5232cf37db6d16f9fc85687c65800ee78ee64ab1f20a9dd0000d1\": container with ID starting with 73a2686a03a5232cf37db6d16f9fc85687c65800ee78ee64ab1f20a9dd0000d1 not found: ID does not exist" Jul 29 04:44:38.178088 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:38.178061 2579 scope.go:117] "RemoveContainer" containerID="3b2a0c70fdf5e5df6d51d9f02d4e5f5cdd690b7bf22d9fd1b41e475279e43f76" Jul 29 04:44:38.184154 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:38.184128 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-994302ee-8669496594-mrz46"] Jul 29 04:44:38.186646 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:38.186623 2579 scope.go:117] "RemoveContainer" containerID="3b2a0c70fdf5e5df6d51d9f02d4e5f5cdd690b7bf22d9fd1b41e475279e43f76" Jul 29 04:44:38.186907 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:44:38.186888 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b2a0c70fdf5e5df6d51d9f02d4e5f5cdd690b7bf22d9fd1b41e475279e43f76\": container with ID starting with 3b2a0c70fdf5e5df6d51d9f02d4e5f5cdd690b7bf22d9fd1b41e475279e43f76 not found: ID does not exist" containerID="3b2a0c70fdf5e5df6d51d9f02d4e5f5cdd690b7bf22d9fd1b41e475279e43f76" Jul 29 04:44:38.186953 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:38.186916 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b2a0c70fdf5e5df6d51d9f02d4e5f5cdd690b7bf22d9fd1b41e475279e43f76"} err="failed to get container status \"3b2a0c70fdf5e5df6d51d9f02d4e5f5cdd690b7bf22d9fd1b41e475279e43f76\": rpc error: code = NotFound desc = could not find container \"3b2a0c70fdf5e5df6d51d9f02d4e5f5cdd690b7bf22d9fd1b41e475279e43f76\": container with ID starting with 3b2a0c70fdf5e5df6d51d9f02d4e5f5cdd690b7bf22d9fd1b41e475279e43f76 not found: ID does not exist" Jul 29 04:44:38.187429 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:38.187412 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-994302ee-8669496594-mrz46"] Jul 29 04:44:38.196461 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:38.196439 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-994302ee-7478b885b8-fjxn2"] Jul 29 04:44:38.199956 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:38.199932 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-994302ee-7478b885b8-fjxn2"] Jul 29 04:44:39.591240 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:39.591191 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="100a7a35-72de-4640-8273-47572d909dbd" path="/var/lib/kubelet/pods/100a7a35-72de-4640-8273-47572d909dbd/volumes" Jul 29 04:44:39.591596 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:44:39.591569 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4b171c8-e2b2-4d45-81df-abed04234b5b" path="/var/lib/kubelet/pods/a4b171c8-e2b2-4d45-81df-abed04234b5b/volumes" Jul 29 04:46:57.212759 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:46:57.212720 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e021d8-bcf5c58d-z4b5p"] Jul 29 04:46:57.213275 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:46:57.213124 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4b171c8-e2b2-4d45-81df-abed04234b5b" containerName="payload-pre-processing" Jul 29 04:46:57.213275 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:46:57.213137 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4b171c8-e2b2-4d45-81df-abed04234b5b" containerName="payload-pre-processing" Jul 29 04:46:57.213275 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:46:57.213147 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="100a7a35-72de-4640-8273-47572d909dbd" containerName="payload-processing" Jul 29 04:46:57.213275 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:46:57.213155 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="100a7a35-72de-4640-8273-47572d909dbd" containerName="payload-processing" Jul 29 04:46:57.213275 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:46:57.213234 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="100a7a35-72de-4640-8273-47572d909dbd" containerName="payload-processing" Jul 29 04:46:57.213275 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:46:57.213249 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4b171c8-e2b2-4d45-81df-abed04234b5b" containerName="payload-pre-processing" Jul 29 04:46:57.216150 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:46:57.216133 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e021d8-bcf5c58d-z4b5p" Jul 29 04:46:57.218480 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:46:57.218459 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e021d8-dockercfg-vsf6f\"" Jul 29 04:46:57.218575 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:46:57.218459 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e021d8\"" Jul 29 04:46:57.222911 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:46:57.222885 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e021d8-bcf5c58d-z4b5p"] Jul 29 04:46:57.344475 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:46:57.344434 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/338f491a-5879-407b-99e9-214fca2be3c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e021d8-bcf5c58d-z4b5p\" (UID: \"338f491a-5879-407b-99e9-214fca2be3c8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e021d8-bcf5c58d-z4b5p" Jul 29 04:46:57.344659 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:46:57.344538 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mglp5\" (UniqueName: \"kubernetes.io/projected/338f491a-5879-407b-99e9-214fca2be3c8-kube-api-access-mglp5\") pod \"payload-pre-processing-e2e-inf-a-e021d8-bcf5c58d-z4b5p\" (UID: \"338f491a-5879-407b-99e9-214fca2be3c8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e021d8-bcf5c58d-z4b5p" Jul 29 04:46:57.445560 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:46:57.445515 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/338f491a-5879-407b-99e9-214fca2be3c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e021d8-bcf5c58d-z4b5p\" (UID: \"338f491a-5879-407b-99e9-214fca2be3c8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e021d8-bcf5c58d-z4b5p" Jul 29 04:46:57.445783 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:46:57.445614 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mglp5\" (UniqueName: \"kubernetes.io/projected/338f491a-5879-407b-99e9-214fca2be3c8-kube-api-access-mglp5\") pod \"payload-pre-processing-e2e-inf-a-e021d8-bcf5c58d-z4b5p\" (UID: \"338f491a-5879-407b-99e9-214fca2be3c8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e021d8-bcf5c58d-z4b5p" Jul 29 04:46:57.446448 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:46:57.446420 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/338f491a-5879-407b-99e9-214fca2be3c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e021d8-bcf5c58d-z4b5p\" (UID: \"338f491a-5879-407b-99e9-214fca2be3c8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e021d8-bcf5c58d-z4b5p" Jul 29 04:46:57.453171 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:46:57.453140 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mglp5\" (UniqueName: \"kubernetes.io/projected/338f491a-5879-407b-99e9-214fca2be3c8-kube-api-access-mglp5\") pod \"payload-pre-processing-e2e-inf-a-e021d8-bcf5c58d-z4b5p\" (UID: \"338f491a-5879-407b-99e9-214fca2be3c8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e021d8-bcf5c58d-z4b5p" Jul 29 04:46:57.528486 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:46:57.528385 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e021d8-bcf5c58d-z4b5p" Jul 29 04:46:57.667457 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:46:57.667424 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e021d8-bcf5c58d-z4b5p"] Jul 29 04:46:57.671168 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:46:57.671140 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod338f491a_5879_407b_99e9_214fca2be3c8.slice/crio-96d9a2e1124bc72402358a1c4ea1a8a1ff33eee147c501d5fc0e9241e02934f0 WatchSource:0}: Error finding container 96d9a2e1124bc72402358a1c4ea1a8a1ff33eee147c501d5fc0e9241e02934f0: Status 404 returned error can't find the container with id 96d9a2e1124bc72402358a1c4ea1a8a1ff33eee147c501d5fc0e9241e02934f0 Jul 29 04:46:57.848577 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:46:57.848468 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e021d8-bcf5c58d-z4b5p" event={"ID":"338f491a-5879-407b-99e9-214fca2be3c8","Type":"ContainerStarted","Data":"13adba281179c6b21990ad422fec1dd61bdd325265b354c009466231f0c00e4a"} Jul 29 04:46:57.848577 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:46:57.848562 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e021d8-bcf5c58d-z4b5p" event={"ID":"338f491a-5879-407b-99e9-214fca2be3c8","Type":"ContainerStarted","Data":"96d9a2e1124bc72402358a1c4ea1a8a1ff33eee147c501d5fc0e9241e02934f0"} Jul 29 04:46:57.848811 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:46:57.848604 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e021d8-bcf5c58d-z4b5p" Jul 29 04:46:57.864615 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:46:57.864564 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e021d8-bcf5c58d-z4b5p" podStartSLOduration=0.864537814 podStartE2EDuration="864.537814ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:46:57.862119677 +0000 UTC m=+2598.721494393" watchObservedRunningTime="2026-07-29 04:46:57.864537814 +0000 UTC m=+2598.723912539" Jul 29 04:47:08.856237 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:08.856179 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e021d8-bcf5c58d-z4b5p" Jul 29 04:47:25.469831 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:25.469794 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ff425c-746568894d-h5pxz"] Jul 29 04:47:25.473246 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:25.473229 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff425c-746568894d-h5pxz" Jul 29 04:47:25.476078 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:25.476055 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ff425c-dockercfg-xdxjd\"" Jul 29 04:47:25.476298 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:25.476277 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ff425c\"" Jul 29 04:47:25.488734 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:25.488709 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ff425c-746568894d-h5pxz"] Jul 29 04:47:25.497830 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:25.497802 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ff425c-6dbf474fd4-2fdqs"] Jul 29 04:47:25.502152 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:25.502117 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ff425c-6dbf474fd4-2fdqs" Jul 29 04:47:25.510222 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:25.510182 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ff425c-6dbf474fd4-2fdqs"] Jul 29 04:47:25.595073 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:25.595024 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdwv7\" (UniqueName: \"kubernetes.io/projected/94c09f7b-3c02-4588-9f60-d68b51ff8a91-kube-api-access-tdwv7\") pod \"payload-pre-processing-e2e-inf-b-ff425c-746568894d-h5pxz\" (UID: \"94c09f7b-3c02-4588-9f60-d68b51ff8a91\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff425c-746568894d-h5pxz" Jul 29 04:47:25.595265 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:25.595162 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94c09f7b-3c02-4588-9f60-d68b51ff8a91-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ff425c-746568894d-h5pxz\" (UID: \"94c09f7b-3c02-4588-9f60-d68b51ff8a91\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff425c-746568894d-h5pxz" Jul 29 04:47:25.595265 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:25.595217 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a3d493a-6bca-41ce-a9b9-24bf4a256858-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ff425c-6dbf474fd4-2fdqs\" (UID: \"6a3d493a-6bca-41ce-a9b9-24bf4a256858\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ff425c-6dbf474fd4-2fdqs" Jul 29 04:47:25.595265 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:25.595250 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjhmv\" (UniqueName: \"kubernetes.io/projected/6a3d493a-6bca-41ce-a9b9-24bf4a256858-kube-api-access-jjhmv\") pod \"payload-processing-e2e-inf-b-ff425c-6dbf474fd4-2fdqs\" (UID: \"6a3d493a-6bca-41ce-a9b9-24bf4a256858\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ff425c-6dbf474fd4-2fdqs" Jul 29 04:47:25.696282 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:25.696241 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94c09f7b-3c02-4588-9f60-d68b51ff8a91-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ff425c-746568894d-h5pxz\" (UID: \"94c09f7b-3c02-4588-9f60-d68b51ff8a91\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff425c-746568894d-h5pxz" Jul 29 04:47:25.696282 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:25.696287 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a3d493a-6bca-41ce-a9b9-24bf4a256858-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ff425c-6dbf474fd4-2fdqs\" (UID: \"6a3d493a-6bca-41ce-a9b9-24bf4a256858\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ff425c-6dbf474fd4-2fdqs" Jul 29 04:47:25.696570 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:25.696309 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjhmv\" (UniqueName: \"kubernetes.io/projected/6a3d493a-6bca-41ce-a9b9-24bf4a256858-kube-api-access-jjhmv\") pod \"payload-processing-e2e-inf-b-ff425c-6dbf474fd4-2fdqs\" (UID: \"6a3d493a-6bca-41ce-a9b9-24bf4a256858\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ff425c-6dbf474fd4-2fdqs" Jul 29 04:47:25.696570 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:25.696339 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdwv7\" (UniqueName: \"kubernetes.io/projected/94c09f7b-3c02-4588-9f60-d68b51ff8a91-kube-api-access-tdwv7\") pod \"payload-pre-processing-e2e-inf-b-ff425c-746568894d-h5pxz\" (UID: \"94c09f7b-3c02-4588-9f60-d68b51ff8a91\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff425c-746568894d-h5pxz" Jul 29 04:47:25.697119 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:25.697098 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a3d493a-6bca-41ce-a9b9-24bf4a256858-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ff425c-6dbf474fd4-2fdqs\" (UID: \"6a3d493a-6bca-41ce-a9b9-24bf4a256858\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ff425c-6dbf474fd4-2fdqs" Jul 29 04:47:25.697160 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:25.697131 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94c09f7b-3c02-4588-9f60-d68b51ff8a91-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ff425c-746568894d-h5pxz\" (UID: \"94c09f7b-3c02-4588-9f60-d68b51ff8a91\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff425c-746568894d-h5pxz" Jul 29 04:47:25.704435 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:25.704407 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjhmv\" (UniqueName: \"kubernetes.io/projected/6a3d493a-6bca-41ce-a9b9-24bf4a256858-kube-api-access-jjhmv\") pod \"payload-processing-e2e-inf-b-ff425c-6dbf474fd4-2fdqs\" (UID: \"6a3d493a-6bca-41ce-a9b9-24bf4a256858\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ff425c-6dbf474fd4-2fdqs" Jul 29 04:47:25.704552 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:25.704506 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdwv7\" (UniqueName: \"kubernetes.io/projected/94c09f7b-3c02-4588-9f60-d68b51ff8a91-kube-api-access-tdwv7\") pod \"payload-pre-processing-e2e-inf-b-ff425c-746568894d-h5pxz\" (UID: \"94c09f7b-3c02-4588-9f60-d68b51ff8a91\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff425c-746568894d-h5pxz" Jul 29 04:47:25.783517 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:25.783433 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff425c-746568894d-h5pxz" Jul 29 04:47:25.813647 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:25.813603 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ff425c-6dbf474fd4-2fdqs" Jul 29 04:47:25.935926 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:25.935898 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ff425c-746568894d-h5pxz"] Jul 29 04:47:25.938578 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:47:25.938545 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c09f7b_3c02_4588_9f60_d68b51ff8a91.slice/crio-a73b70ceb2638f2eaae9868e2c7cc0f651b8fec3ba437a95ad8d46dbaec247b8 WatchSource:0}: Error finding container a73b70ceb2638f2eaae9868e2c7cc0f651b8fec3ba437a95ad8d46dbaec247b8: Status 404 returned error can't find the container with id a73b70ceb2638f2eaae9868e2c7cc0f651b8fec3ba437a95ad8d46dbaec247b8 Jul 29 04:47:25.971096 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:25.971068 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ff425c-6dbf474fd4-2fdqs"] Jul 29 04:47:25.973835 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:47:25.973810 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a3d493a_6bca_41ce_a9b9_24bf4a256858.slice/crio-e5531902ae7671102f1500a48312bf6e200e21dee0d6befc8c215adbdd12e235 WatchSource:0}: Error finding container e5531902ae7671102f1500a48312bf6e200e21dee0d6befc8c215adbdd12e235: Status 404 returned error can't find the container with id e5531902ae7671102f1500a48312bf6e200e21dee0d6befc8c215adbdd12e235 Jul 29 04:47:25.988067 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:25.988036 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff425c-746568894d-h5pxz" event={"ID":"94c09f7b-3c02-4588-9f60-d68b51ff8a91","Type":"ContainerStarted","Data":"a73b70ceb2638f2eaae9868e2c7cc0f651b8fec3ba437a95ad8d46dbaec247b8"} Jul 29 04:47:25.990486 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:25.990422 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ff425c-6dbf474fd4-2fdqs" event={"ID":"6a3d493a-6bca-41ce-a9b9-24bf4a256858","Type":"ContainerStarted","Data":"e5531902ae7671102f1500a48312bf6e200e21dee0d6befc8c215adbdd12e235"} Jul 29 04:47:26.996482 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:26.996449 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ff425c-6dbf474fd4-2fdqs" event={"ID":"6a3d493a-6bca-41ce-a9b9-24bf4a256858","Type":"ContainerStarted","Data":"304f3a925887f4d0030cfac4eeb3ff7bf899576a2f0949917064f8eee27ab315"} Jul 29 04:47:26.996945 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:26.996510 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ff425c-6dbf474fd4-2fdqs" Jul 29 04:47:26.997907 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:26.997886 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff425c-746568894d-h5pxz" event={"ID":"94c09f7b-3c02-4588-9f60-d68b51ff8a91","Type":"ContainerStarted","Data":"d7214142692f3fc90d229213af084b5777cfa16a0b8ac9a953ab104d27161fd9"} Jul 29 04:47:26.997993 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:26.997927 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff425c-746568894d-h5pxz" Jul 29 04:47:27.012131 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:27.012076 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ff425c-6dbf474fd4-2fdqs" podStartSLOduration=2.012058192 podStartE2EDuration="2.012058192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:47:27.011113086 +0000 UTC m=+2627.870487908" watchObservedRunningTime="2026-07-29 04:47:27.012058192 +0000 UTC m=+2627.871432917" Jul 29 04:47:27.026370 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:27.026318 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff425c-746568894d-h5pxz" podStartSLOduration=2.026297523 podStartE2EDuration="2.026297523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:47:27.025520715 +0000 UTC m=+2627.884895453" watchObservedRunningTime="2026-07-29 04:47:27.026297523 +0000 UTC m=+2627.885672248" Jul 29 04:47:38.005921 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:38.005866 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff425c-746568894d-h5pxz" podUID="94c09f7b-3c02-4588-9f60-d68b51ff8a91" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.63:9004: connect: connection refused" Jul 29 04:47:38.006484 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:38.006280 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ff425c-6dbf474fd4-2fdqs" Jul 29 04:47:48.004949 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:47:48.004918 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff425c-746568894d-h5pxz" Jul 29 04:48:32.926918 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:32.926838 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg"] Jul 29 04:48:32.930473 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:32.930452 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" Jul 29 04:48:32.933444 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:32.933418 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ff425c\"/\"openshift-service-ca.crt\"" Jul 29 04:48:32.933444 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:32.933432 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ff425c\"/\"test-model-ff425c-kserve-self-signed-certs\"" Jul 29 04:48:32.933444 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:32.933442 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ff425c\"/\"default-dockercfg-z4594\"" Jul 29 04:48:32.933689 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:32.933455 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ff425c\"/\"kube-root-ca.crt\"" Jul 29 04:48:32.940037 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:32.940014 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg"] Jul 29 04:48:33.002888 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:33.002856 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da47b086-ae65-497f-b194-20aa9f589cfb-tmp-dir\") pod \"test-model-ff425c-kserve-7996659f8f-znlbg\" (UID: \"da47b086-ae65-497f-b194-20aa9f589cfb\") " pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" Jul 29 04:48:33.003065 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:33.002913 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da47b086-ae65-497f-b194-20aa9f589cfb-home\") pod \"test-model-ff425c-kserve-7996659f8f-znlbg\" (UID: \"da47b086-ae65-497f-b194-20aa9f589cfb\") " pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" Jul 29 04:48:33.003065 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:33.002934 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da47b086-ae65-497f-b194-20aa9f589cfb-model-cache\") pod \"test-model-ff425c-kserve-7996659f8f-znlbg\" (UID: \"da47b086-ae65-497f-b194-20aa9f589cfb\") " pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" Jul 29 04:48:33.003065 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:33.002952 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hsnx\" (UniqueName: \"kubernetes.io/projected/da47b086-ae65-497f-b194-20aa9f589cfb-kube-api-access-7hsnx\") pod \"test-model-ff425c-kserve-7996659f8f-znlbg\" (UID: \"da47b086-ae65-497f-b194-20aa9f589cfb\") " pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" Jul 29 04:48:33.003065 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:33.002976 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da47b086-ae65-497f-b194-20aa9f589cfb-tls-certs\") pod \"test-model-ff425c-kserve-7996659f8f-znlbg\" (UID: \"da47b086-ae65-497f-b194-20aa9f589cfb\") " pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" Jul 29 04:48:33.003065 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:33.003005 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da47b086-ae65-497f-b194-20aa9f589cfb-dshm\") pod \"test-model-ff425c-kserve-7996659f8f-znlbg\" (UID: \"da47b086-ae65-497f-b194-20aa9f589cfb\") " pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" Jul 29 04:48:33.103561 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:33.103523 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da47b086-ae65-497f-b194-20aa9f589cfb-tmp-dir\") pod \"test-model-ff425c-kserve-7996659f8f-znlbg\" (UID: \"da47b086-ae65-497f-b194-20aa9f589cfb\") " pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" Jul 29 04:48:33.103753 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:33.103579 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da47b086-ae65-497f-b194-20aa9f589cfb-home\") pod \"test-model-ff425c-kserve-7996659f8f-znlbg\" (UID: \"da47b086-ae65-497f-b194-20aa9f589cfb\") " pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" Jul 29 04:48:33.103753 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:33.103601 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da47b086-ae65-497f-b194-20aa9f589cfb-model-cache\") pod \"test-model-ff425c-kserve-7996659f8f-znlbg\" (UID: \"da47b086-ae65-497f-b194-20aa9f589cfb\") " pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" Jul 29 04:48:33.103753 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:33.103617 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hsnx\" (UniqueName: \"kubernetes.io/projected/da47b086-ae65-497f-b194-20aa9f589cfb-kube-api-access-7hsnx\") pod \"test-model-ff425c-kserve-7996659f8f-znlbg\" (UID: \"da47b086-ae65-497f-b194-20aa9f589cfb\") " pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" Jul 29 04:48:33.103753 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:33.103642 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da47b086-ae65-497f-b194-20aa9f589cfb-tls-certs\") pod \"test-model-ff425c-kserve-7996659f8f-znlbg\" (UID: \"da47b086-ae65-497f-b194-20aa9f589cfb\") " pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" Jul 29 04:48:33.103753 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:33.103668 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da47b086-ae65-497f-b194-20aa9f589cfb-dshm\") pod \"test-model-ff425c-kserve-7996659f8f-znlbg\" (UID: \"da47b086-ae65-497f-b194-20aa9f589cfb\") " pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" Jul 29 04:48:33.104031 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:33.103968 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da47b086-ae65-497f-b194-20aa9f589cfb-tmp-dir\") pod \"test-model-ff425c-kserve-7996659f8f-znlbg\" (UID: \"da47b086-ae65-497f-b194-20aa9f589cfb\") " pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" Jul 29 04:48:33.104031 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:33.104010 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da47b086-ae65-497f-b194-20aa9f589cfb-model-cache\") pod \"test-model-ff425c-kserve-7996659f8f-znlbg\" (UID: \"da47b086-ae65-497f-b194-20aa9f589cfb\") " pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" Jul 29 04:48:33.104124 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:33.104053 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da47b086-ae65-497f-b194-20aa9f589cfb-home\") pod \"test-model-ff425c-kserve-7996659f8f-znlbg\" (UID: \"da47b086-ae65-497f-b194-20aa9f589cfb\") " pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" Jul 29 04:48:33.105918 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:33.105896 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da47b086-ae65-497f-b194-20aa9f589cfb-dshm\") pod \"test-model-ff425c-kserve-7996659f8f-znlbg\" (UID: \"da47b086-ae65-497f-b194-20aa9f589cfb\") " pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" Jul 29 04:48:33.106280 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:33.106263 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da47b086-ae65-497f-b194-20aa9f589cfb-tls-certs\") pod \"test-model-ff425c-kserve-7996659f8f-znlbg\" (UID: \"da47b086-ae65-497f-b194-20aa9f589cfb\") " pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" Jul 29 04:48:33.110664 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:33.110642 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hsnx\" (UniqueName: \"kubernetes.io/projected/da47b086-ae65-497f-b194-20aa9f589cfb-kube-api-access-7hsnx\") pod \"test-model-ff425c-kserve-7996659f8f-znlbg\" (UID: \"da47b086-ae65-497f-b194-20aa9f589cfb\") " pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" Jul 29 04:48:33.240724 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:33.240686 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" Jul 29 04:48:33.376002 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:33.375975 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg"] Jul 29 04:48:33.377533 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:48:33.377503 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda47b086_ae65_497f_b194_20aa9f589cfb.slice/crio-3afb554910269dbcd5eb86a1c8de74c10ec26d06ae9240421f77c3dcb4085f74 WatchSource:0}: Error finding container 3afb554910269dbcd5eb86a1c8de74c10ec26d06ae9240421f77c3dcb4085f74: Status 404 returned error can't find the container with id 3afb554910269dbcd5eb86a1c8de74c10ec26d06ae9240421f77c3dcb4085f74 Jul 29 04:48:34.332116 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:34.332081 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" event={"ID":"da47b086-ae65-497f-b194-20aa9f589cfb","Type":"ContainerStarted","Data":"283b7a25b6386877b743decac00fe8e334a0fe472302d7f6c6d4e89718fd603c"} Jul 29 04:48:34.332116 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:34.332121 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" event={"ID":"da47b086-ae65-497f-b194-20aa9f589cfb","Type":"ContainerStarted","Data":"3afb554910269dbcd5eb86a1c8de74c10ec26d06ae9240421f77c3dcb4085f74"} Jul 29 04:48:43.241600 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:43.241560 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" Jul 29 04:48:43.242014 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:43.241646 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" Jul 29 04:48:43.254185 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:43.254162 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" Jul 29 04:48:43.270515 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:43.270462 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" podStartSLOduration=11.27044783 podStartE2EDuration="11.27044783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:48:34.347757659 +0000 UTC m=+2695.207132383" watchObservedRunningTime="2026-07-29 04:48:43.27044783 +0000 UTC m=+2704.129822553" Jul 29 04:48:43.387658 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:48:43.387631 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" Jul 29 04:49:15.208075 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:15.208042 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg"] Jul 29 04:49:25.915523 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:25.915484 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e021d8-bcf5c58d-z4b5p"] Jul 29 04:49:25.915964 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:25.915728 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e021d8-bcf5c58d-z4b5p" podUID="338f491a-5879-407b-99e9-214fca2be3c8" containerName="payload-pre-processing" containerID="cri-o://13adba281179c6b21990ad422fec1dd61bdd325265b354c009466231f0c00e4a" gracePeriod=30 Jul 29 04:49:26.178998 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:26.178970 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e021d8-bcf5c58d-z4b5p" Jul 29 04:49:26.192425 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:26.192399 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mglp5\" (UniqueName: \"kubernetes.io/projected/338f491a-5879-407b-99e9-214fca2be3c8-kube-api-access-mglp5\") pod \"338f491a-5879-407b-99e9-214fca2be3c8\" (UID: \"338f491a-5879-407b-99e9-214fca2be3c8\") " Jul 29 04:49:26.192559 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:26.192433 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/338f491a-5879-407b-99e9-214fca2be3c8-plugins-config-volume\") pod \"338f491a-5879-407b-99e9-214fca2be3c8\" (UID: \"338f491a-5879-407b-99e9-214fca2be3c8\") " Jul 29 04:49:26.192820 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:26.192795 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/338f491a-5879-407b-99e9-214fca2be3c8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "338f491a-5879-407b-99e9-214fca2be3c8" (UID: "338f491a-5879-407b-99e9-214fca2be3c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 04:49:26.194497 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:26.194455 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/338f491a-5879-407b-99e9-214fca2be3c8-kube-api-access-mglp5" (OuterVolumeSpecName: "kube-api-access-mglp5") pod "338f491a-5879-407b-99e9-214fca2be3c8" (UID: "338f491a-5879-407b-99e9-214fca2be3c8"). InnerVolumeSpecName "kube-api-access-mglp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:49:26.293735 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:26.293697 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mglp5\" (UniqueName: \"kubernetes.io/projected/338f491a-5879-407b-99e9-214fca2be3c8-kube-api-access-mglp5\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:49:26.293735 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:26.293730 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/338f491a-5879-407b-99e9-214fca2be3c8-plugins-config-volume\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:49:26.587643 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:26.587548 2579 generic.go:356] "Generic (PLEG): container finished" podID="338f491a-5879-407b-99e9-214fca2be3c8" containerID="13adba281179c6b21990ad422fec1dd61bdd325265b354c009466231f0c00e4a" exitCode=0 Jul 29 04:49:26.587643 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:26.587589 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e021d8-bcf5c58d-z4b5p" Jul 29 04:49:26.587891 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:26.587646 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e021d8-bcf5c58d-z4b5p" event={"ID":"338f491a-5879-407b-99e9-214fca2be3c8","Type":"ContainerDied","Data":"13adba281179c6b21990ad422fec1dd61bdd325265b354c009466231f0c00e4a"} Jul 29 04:49:26.587891 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:26.587702 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e021d8-bcf5c58d-z4b5p" event={"ID":"338f491a-5879-407b-99e9-214fca2be3c8","Type":"ContainerDied","Data":"96d9a2e1124bc72402358a1c4ea1a8a1ff33eee147c501d5fc0e9241e02934f0"} Jul 29 04:49:26.587891 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:26.587730 2579 scope.go:117] "RemoveContainer" containerID="13adba281179c6b21990ad422fec1dd61bdd325265b354c009466231f0c00e4a" Jul 29 04:49:26.597679 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:26.597660 2579 scope.go:117] "RemoveContainer" containerID="13adba281179c6b21990ad422fec1dd61bdd325265b354c009466231f0c00e4a" Jul 29 04:49:26.597979 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:49:26.597962 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13adba281179c6b21990ad422fec1dd61bdd325265b354c009466231f0c00e4a\": container with ID starting with 13adba281179c6b21990ad422fec1dd61bdd325265b354c009466231f0c00e4a not found: ID does not exist" containerID="13adba281179c6b21990ad422fec1dd61bdd325265b354c009466231f0c00e4a" Jul 29 04:49:26.598015 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:26.597987 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13adba281179c6b21990ad422fec1dd61bdd325265b354c009466231f0c00e4a"} err="failed to get container status \"13adba281179c6b21990ad422fec1dd61bdd325265b354c009466231f0c00e4a\": rpc error: code = NotFound desc = could not find container \"13adba281179c6b21990ad422fec1dd61bdd325265b354c009466231f0c00e4a\": container with ID starting with 13adba281179c6b21990ad422fec1dd61bdd325265b354c009466231f0c00e4a not found: ID does not exist" Jul 29 04:49:26.609086 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:26.609059 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e021d8-bcf5c58d-z4b5p"] Jul 29 04:49:26.616427 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:26.616400 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e021d8-bcf5c58d-z4b5p"] Jul 29 04:49:27.591647 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:27.591608 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="338f491a-5879-407b-99e9-214fca2be3c8" path="/var/lib/kubelet/pods/338f491a-5879-407b-99e9-214fca2be3c8/volumes" Jul 29 04:49:30.227217 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:30.227156 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" podUID="da47b086-ae65-497f-b194-20aa9f589cfb" containerName="main" containerID="cri-o://283b7a25b6386877b743decac00fe8e334a0fe472302d7f6c6d4e89718fd603c" gracePeriod=45 Jul 29 04:49:30.485623 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:30.485561 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" Jul 29 04:49:30.529807 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:30.529778 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da47b086-ae65-497f-b194-20aa9f589cfb-model-cache\") pod \"da47b086-ae65-497f-b194-20aa9f589cfb\" (UID: \"da47b086-ae65-497f-b194-20aa9f589cfb\") " Jul 29 04:49:30.530010 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:30.529826 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da47b086-ae65-497f-b194-20aa9f589cfb-tls-certs\") pod \"da47b086-ae65-497f-b194-20aa9f589cfb\" (UID: \"da47b086-ae65-497f-b194-20aa9f589cfb\") " Jul 29 04:49:30.530010 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:30.529880 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da47b086-ae65-497f-b194-20aa9f589cfb-dshm\") pod \"da47b086-ae65-497f-b194-20aa9f589cfb\" (UID: \"da47b086-ae65-497f-b194-20aa9f589cfb\") " Jul 29 04:49:30.530010 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:30.529914 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da47b086-ae65-497f-b194-20aa9f589cfb-tmp-dir\") pod \"da47b086-ae65-497f-b194-20aa9f589cfb\" (UID: \"da47b086-ae65-497f-b194-20aa9f589cfb\") " Jul 29 04:49:30.530010 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:30.529954 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da47b086-ae65-497f-b194-20aa9f589cfb-home\") pod \"da47b086-ae65-497f-b194-20aa9f589cfb\" (UID: \"da47b086-ae65-497f-b194-20aa9f589cfb\") " Jul 29 04:49:30.530010 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:30.530001 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hsnx\" (UniqueName: \"kubernetes.io/projected/da47b086-ae65-497f-b194-20aa9f589cfb-kube-api-access-7hsnx\") pod \"da47b086-ae65-497f-b194-20aa9f589cfb\" (UID: \"da47b086-ae65-497f-b194-20aa9f589cfb\") " Jul 29 04:49:30.530318 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:30.530101 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da47b086-ae65-497f-b194-20aa9f589cfb-model-cache" (OuterVolumeSpecName: "model-cache") pod "da47b086-ae65-497f-b194-20aa9f589cfb" (UID: "da47b086-ae65-497f-b194-20aa9f589cfb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 04:49:30.530449 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:30.530389 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da47b086-ae65-497f-b194-20aa9f589cfb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "da47b086-ae65-497f-b194-20aa9f589cfb" (UID: "da47b086-ae65-497f-b194-20aa9f589cfb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 04:49:30.530449 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:30.530401 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da47b086-ae65-497f-b194-20aa9f589cfb-model-cache\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:49:30.530670 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:30.530606 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da47b086-ae65-497f-b194-20aa9f589cfb-home" (OuterVolumeSpecName: "home") pod "da47b086-ae65-497f-b194-20aa9f589cfb" (UID: "da47b086-ae65-497f-b194-20aa9f589cfb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 04:49:30.532666 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:30.532636 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da47b086-ae65-497f-b194-20aa9f589cfb-kube-api-access-7hsnx" (OuterVolumeSpecName: "kube-api-access-7hsnx") pod "da47b086-ae65-497f-b194-20aa9f589cfb" (UID: "da47b086-ae65-497f-b194-20aa9f589cfb"). InnerVolumeSpecName "kube-api-access-7hsnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:49:30.532987 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:30.532964 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da47b086-ae65-497f-b194-20aa9f589cfb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "da47b086-ae65-497f-b194-20aa9f589cfb" (UID: "da47b086-ae65-497f-b194-20aa9f589cfb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 04:49:30.533128 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:30.533111 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da47b086-ae65-497f-b194-20aa9f589cfb-dshm" (OuterVolumeSpecName: "dshm") pod "da47b086-ae65-497f-b194-20aa9f589cfb" (UID: "da47b086-ae65-497f-b194-20aa9f589cfb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 04:49:30.607551 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:30.607513 2579 generic.go:356] "Generic (PLEG): container finished" podID="da47b086-ae65-497f-b194-20aa9f589cfb" containerID="283b7a25b6386877b743decac00fe8e334a0fe472302d7f6c6d4e89718fd603c" exitCode=0 Jul 29 04:49:30.607551 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:30.607552 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" Jul 29 04:49:30.607767 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:30.607580 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" event={"ID":"da47b086-ae65-497f-b194-20aa9f589cfb","Type":"ContainerDied","Data":"283b7a25b6386877b743decac00fe8e334a0fe472302d7f6c6d4e89718fd603c"} Jul 29 04:49:30.607767 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:30.607617 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg" event={"ID":"da47b086-ae65-497f-b194-20aa9f589cfb","Type":"ContainerDied","Data":"3afb554910269dbcd5eb86a1c8de74c10ec26d06ae9240421f77c3dcb4085f74"} Jul 29 04:49:30.607767 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:30.607634 2579 scope.go:117] "RemoveContainer" containerID="283b7a25b6386877b743decac00fe8e334a0fe472302d7f6c6d4e89718fd603c" Jul 29 04:49:30.617138 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:30.617088 2579 scope.go:117] "RemoveContainer" containerID="283b7a25b6386877b743decac00fe8e334a0fe472302d7f6c6d4e89718fd603c" Jul 29 04:49:30.617385 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:49:30.617365 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"283b7a25b6386877b743decac00fe8e334a0fe472302d7f6c6d4e89718fd603c\": container with ID starting with 283b7a25b6386877b743decac00fe8e334a0fe472302d7f6c6d4e89718fd603c not found: ID does not exist" containerID="283b7a25b6386877b743decac00fe8e334a0fe472302d7f6c6d4e89718fd603c" Jul 29 04:49:30.617464 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:30.617393 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"283b7a25b6386877b743decac00fe8e334a0fe472302d7f6c6d4e89718fd603c"} err="failed to get container status \"283b7a25b6386877b743decac00fe8e334a0fe472302d7f6c6d4e89718fd603c\": rpc error: code = NotFound desc = could not find container \"283b7a25b6386877b743decac00fe8e334a0fe472302d7f6c6d4e89718fd603c\": container with ID starting with 283b7a25b6386877b743decac00fe8e334a0fe472302d7f6c6d4e89718fd603c not found: ID does not exist" Jul 29 04:49:30.628366 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:30.628343 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg"] Jul 29 04:49:30.631879 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:30.631826 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da47b086-ae65-497f-b194-20aa9f589cfb-tmp-dir\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:49:30.631879 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:30.631852 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da47b086-ae65-497f-b194-20aa9f589cfb-home\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:49:30.631879 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:30.631866 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hsnx\" (UniqueName: \"kubernetes.io/projected/da47b086-ae65-497f-b194-20aa9f589cfb-kube-api-access-7hsnx\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:49:30.631879 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:30.631880 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da47b086-ae65-497f-b194-20aa9f589cfb-tls-certs\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:49:30.632187 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:30.631894 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da47b086-ae65-497f-b194-20aa9f589cfb-dshm\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:49:30.634391 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:30.634370 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ff425c/test-model-ff425c-kserve-7996659f8f-znlbg"] Jul 29 04:49:31.593275 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:31.593243 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da47b086-ae65-497f-b194-20aa9f589cfb" path="/var/lib/kubelet/pods/da47b086-ae65-497f-b194-20aa9f589cfb/volumes" Jul 29 04:49:43.189459 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.189424 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ff425c-6dbf474fd4-2fdqs"] Jul 29 04:49:43.189982 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.189684 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ff425c-6dbf474fd4-2fdqs" podUID="6a3d493a-6bca-41ce-a9b9-24bf4a256858" containerName="payload-processing" containerID="cri-o://304f3a925887f4d0030cfac4eeb3ff7bf899576a2f0949917064f8eee27ab315" gracePeriod=30 Jul 29 04:49:43.197072 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.197045 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ff425c-746568894d-h5pxz"] Jul 29 04:49:43.197342 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.197295 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff425c-746568894d-h5pxz" podUID="94c09f7b-3c02-4588-9f60-d68b51ff8a91" containerName="payload-pre-processing" containerID="cri-o://d7214142692f3fc90d229213af084b5777cfa16a0b8ac9a953ab104d27161fd9" gracePeriod=30 Jul 29 04:49:43.484390 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.484360 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ff425c-6dbf474fd4-2fdqs" Jul 29 04:49:43.487361 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.487342 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff425c-746568894d-h5pxz" Jul 29 04:49:43.546145 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.546105 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94c09f7b-3c02-4588-9f60-d68b51ff8a91-plugins-config-volume\") pod \"94c09f7b-3c02-4588-9f60-d68b51ff8a91\" (UID: \"94c09f7b-3c02-4588-9f60-d68b51ff8a91\") " Jul 29 04:49:43.546356 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.546163 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdwv7\" (UniqueName: \"kubernetes.io/projected/94c09f7b-3c02-4588-9f60-d68b51ff8a91-kube-api-access-tdwv7\") pod \"94c09f7b-3c02-4588-9f60-d68b51ff8a91\" (UID: \"94c09f7b-3c02-4588-9f60-d68b51ff8a91\") " Jul 29 04:49:43.546356 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.546256 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjhmv\" (UniqueName: \"kubernetes.io/projected/6a3d493a-6bca-41ce-a9b9-24bf4a256858-kube-api-access-jjhmv\") pod \"6a3d493a-6bca-41ce-a9b9-24bf4a256858\" (UID: \"6a3d493a-6bca-41ce-a9b9-24bf4a256858\") " Jul 29 04:49:43.546356 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.546271 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a3d493a-6bca-41ce-a9b9-24bf4a256858-plugins-config-volume\") pod \"6a3d493a-6bca-41ce-a9b9-24bf4a256858\" (UID: \"6a3d493a-6bca-41ce-a9b9-24bf4a256858\") " Jul 29 04:49:43.546646 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.546609 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94c09f7b-3c02-4588-9f60-d68b51ff8a91-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "94c09f7b-3c02-4588-9f60-d68b51ff8a91" (UID: "94c09f7b-3c02-4588-9f60-d68b51ff8a91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 04:49:43.546751 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.546724 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a3d493a-6bca-41ce-a9b9-24bf4a256858-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6a3d493a-6bca-41ce-a9b9-24bf4a256858" (UID: "6a3d493a-6bca-41ce-a9b9-24bf4a256858"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 04:49:43.548496 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.548471 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c09f7b-3c02-4588-9f60-d68b51ff8a91-kube-api-access-tdwv7" (OuterVolumeSpecName: "kube-api-access-tdwv7") pod "94c09f7b-3c02-4588-9f60-d68b51ff8a91" (UID: "94c09f7b-3c02-4588-9f60-d68b51ff8a91"). InnerVolumeSpecName "kube-api-access-tdwv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:49:43.548496 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.548482 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a3d493a-6bca-41ce-a9b9-24bf4a256858-kube-api-access-jjhmv" (OuterVolumeSpecName: "kube-api-access-jjhmv") pod "6a3d493a-6bca-41ce-a9b9-24bf4a256858" (UID: "6a3d493a-6bca-41ce-a9b9-24bf4a256858"). InnerVolumeSpecName "kube-api-access-jjhmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:49:43.647889 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.647851 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjhmv\" (UniqueName: \"kubernetes.io/projected/6a3d493a-6bca-41ce-a9b9-24bf4a256858-kube-api-access-jjhmv\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:49:43.647889 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.647878 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a3d493a-6bca-41ce-a9b9-24bf4a256858-plugins-config-volume\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:49:43.647889 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.647889 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94c09f7b-3c02-4588-9f60-d68b51ff8a91-plugins-config-volume\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:49:43.647889 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.647899 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdwv7\" (UniqueName: \"kubernetes.io/projected/94c09f7b-3c02-4588-9f60-d68b51ff8a91-kube-api-access-tdwv7\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:49:43.674525 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.674493 2579 generic.go:356] "Generic (PLEG): container finished" podID="94c09f7b-3c02-4588-9f60-d68b51ff8a91" containerID="d7214142692f3fc90d229213af084b5777cfa16a0b8ac9a953ab104d27161fd9" exitCode=0 Jul 29 04:49:43.674683 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.674544 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff425c-746568894d-h5pxz" Jul 29 04:49:43.674683 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.674542 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff425c-746568894d-h5pxz" event={"ID":"94c09f7b-3c02-4588-9f60-d68b51ff8a91","Type":"ContainerDied","Data":"d7214142692f3fc90d229213af084b5777cfa16a0b8ac9a953ab104d27161fd9"} Jul 29 04:49:43.674683 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.674602 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff425c-746568894d-h5pxz" event={"ID":"94c09f7b-3c02-4588-9f60-d68b51ff8a91","Type":"ContainerDied","Data":"a73b70ceb2638f2eaae9868e2c7cc0f651b8fec3ba437a95ad8d46dbaec247b8"} Jul 29 04:49:43.674683 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.674619 2579 scope.go:117] "RemoveContainer" containerID="d7214142692f3fc90d229213af084b5777cfa16a0b8ac9a953ab104d27161fd9" Jul 29 04:49:43.675996 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.675973 2579 generic.go:356] "Generic (PLEG): container finished" podID="6a3d493a-6bca-41ce-a9b9-24bf4a256858" containerID="304f3a925887f4d0030cfac4eeb3ff7bf899576a2f0949917064f8eee27ab315" exitCode=0 Jul 29 04:49:43.676124 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.676013 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ff425c-6dbf474fd4-2fdqs" Jul 29 04:49:43.676124 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.676056 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ff425c-6dbf474fd4-2fdqs" event={"ID":"6a3d493a-6bca-41ce-a9b9-24bf4a256858","Type":"ContainerDied","Data":"304f3a925887f4d0030cfac4eeb3ff7bf899576a2f0949917064f8eee27ab315"} Jul 29 04:49:43.676124 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.676095 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ff425c-6dbf474fd4-2fdqs" event={"ID":"6a3d493a-6bca-41ce-a9b9-24bf4a256858","Type":"ContainerDied","Data":"e5531902ae7671102f1500a48312bf6e200e21dee0d6befc8c215adbdd12e235"} Jul 29 04:49:43.683106 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.683091 2579 scope.go:117] "RemoveContainer" containerID="d7214142692f3fc90d229213af084b5777cfa16a0b8ac9a953ab104d27161fd9" Jul 29 04:49:43.683435 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:49:43.683416 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7214142692f3fc90d229213af084b5777cfa16a0b8ac9a953ab104d27161fd9\": container with ID starting with d7214142692f3fc90d229213af084b5777cfa16a0b8ac9a953ab104d27161fd9 not found: ID does not exist" containerID="d7214142692f3fc90d229213af084b5777cfa16a0b8ac9a953ab104d27161fd9" Jul 29 04:49:43.683483 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.683442 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7214142692f3fc90d229213af084b5777cfa16a0b8ac9a953ab104d27161fd9"} err="failed to get container status \"d7214142692f3fc90d229213af084b5777cfa16a0b8ac9a953ab104d27161fd9\": rpc error: code = NotFound desc = could not find container \"d7214142692f3fc90d229213af084b5777cfa16a0b8ac9a953ab104d27161fd9\": container with ID starting with d7214142692f3fc90d229213af084b5777cfa16a0b8ac9a953ab104d27161fd9 not found: ID does not exist" Jul 29 04:49:43.683483 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.683458 2579 scope.go:117] "RemoveContainer" containerID="304f3a925887f4d0030cfac4eeb3ff7bf899576a2f0949917064f8eee27ab315" Jul 29 04:49:43.691256 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.691233 2579 scope.go:117] "RemoveContainer" containerID="304f3a925887f4d0030cfac4eeb3ff7bf899576a2f0949917064f8eee27ab315" Jul 29 04:49:43.691583 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:49:43.691554 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"304f3a925887f4d0030cfac4eeb3ff7bf899576a2f0949917064f8eee27ab315\": container with ID starting with 304f3a925887f4d0030cfac4eeb3ff7bf899576a2f0949917064f8eee27ab315 not found: ID does not exist" containerID="304f3a925887f4d0030cfac4eeb3ff7bf899576a2f0949917064f8eee27ab315" Jul 29 04:49:43.691660 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.691589 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"304f3a925887f4d0030cfac4eeb3ff7bf899576a2f0949917064f8eee27ab315"} err="failed to get container status \"304f3a925887f4d0030cfac4eeb3ff7bf899576a2f0949917064f8eee27ab315\": rpc error: code = NotFound desc = could not find container \"304f3a925887f4d0030cfac4eeb3ff7bf899576a2f0949917064f8eee27ab315\": container with ID starting with 304f3a925887f4d0030cfac4eeb3ff7bf899576a2f0949917064f8eee27ab315 not found: ID does not exist" Jul 29 04:49:43.693080 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.693061 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ff425c-6dbf474fd4-2fdqs"] Jul 29 04:49:43.698650 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.698629 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ff425c-6dbf474fd4-2fdqs"] Jul 29 04:49:43.708610 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.708561 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ff425c-746568894d-h5pxz"] Jul 29 04:49:43.712666 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:43.712647 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ff425c-746568894d-h5pxz"] Jul 29 04:49:45.591340 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:45.591310 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a3d493a-6bca-41ce-a9b9-24bf4a256858" path="/var/lib/kubelet/pods/6a3d493a-6bca-41ce-a9b9-24bf4a256858/volumes" Jul 29 04:49:45.591725 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:45.591672 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94c09f7b-3c02-4588-9f60-d68b51ff8a91" path="/var/lib/kubelet/pods/94c09f7b-3c02-4588-9f60-d68b51ff8a91/volumes" Jul 29 04:49:54.400135 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.400101 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-09567c-5764556cf6-rzlqd"] Jul 29 04:49:54.400551 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.400470 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94c09f7b-3c02-4588-9f60-d68b51ff8a91" containerName="payload-pre-processing" Jul 29 04:49:54.400551 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.400481 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="94c09f7b-3c02-4588-9f60-d68b51ff8a91" containerName="payload-pre-processing" Jul 29 04:49:54.400551 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.400489 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da47b086-ae65-497f-b194-20aa9f589cfb" containerName="main" Jul 29 04:49:54.400551 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.400495 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="da47b086-ae65-497f-b194-20aa9f589cfb" containerName="main" Jul 29 04:49:54.400551 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.400505 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a3d493a-6bca-41ce-a9b9-24bf4a256858" containerName="payload-processing" Jul 29 04:49:54.400551 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.400511 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a3d493a-6bca-41ce-a9b9-24bf4a256858" containerName="payload-processing" Jul 29 04:49:54.400551 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.400523 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="338f491a-5879-407b-99e9-214fca2be3c8" containerName="payload-pre-processing" Jul 29 04:49:54.400551 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.400529 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="338f491a-5879-407b-99e9-214fca2be3c8" containerName="payload-pre-processing" Jul 29 04:49:54.400910 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.400587 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="da47b086-ae65-497f-b194-20aa9f589cfb" containerName="main" Jul 29 04:49:54.400910 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.400598 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a3d493a-6bca-41ce-a9b9-24bf4a256858" containerName="payload-processing" Jul 29 04:49:54.400910 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.400605 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="94c09f7b-3c02-4588-9f60-d68b51ff8a91" containerName="payload-pre-processing" Jul 29 04:49:54.400910 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.400611 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="338f491a-5879-407b-99e9-214fca2be3c8" containerName="payload-pre-processing" Jul 29 04:49:54.403548 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.403533 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-09567c-5764556cf6-rzlqd" Jul 29 04:49:54.405973 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.405952 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-09567c\"" Jul 29 04:49:54.406073 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.405952 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-09567c-dockercfg-brs5w\"" Jul 29 04:49:54.413387 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.413365 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-09567c-5764556cf6-rzlqd"] Jul 29 04:49:54.439916 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.439890 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-09567c-68bcf8fd5b-btxlb"] Jul 29 04:49:54.443334 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.443315 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-09567c-68bcf8fd5b-btxlb" Jul 29 04:49:54.455310 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.455279 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-09567c-68bcf8fd5b-btxlb"] Jul 29 04:49:54.539430 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.539394 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27a57217-5de7-4fd5-8a80-81e13f891264-plugins-config-volume\") pod \"payload-processing-e2e-api-a-09567c-68bcf8fd5b-btxlb\" (UID: \"27a57217-5de7-4fd5-8a80-81e13f891264\") " pod="openshift-ingress/payload-processing-e2e-api-a-09567c-68bcf8fd5b-btxlb" Jul 29 04:49:54.539613 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.539463 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-09567c-5764556cf6-rzlqd\" (UID: \"8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-09567c-5764556cf6-rzlqd" Jul 29 04:49:54.539613 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.539507 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x78v\" (UniqueName: \"kubernetes.io/projected/8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2-kube-api-access-8x78v\") pod \"payload-pre-processing-e2e-api-a-09567c-5764556cf6-rzlqd\" (UID: \"8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-09567c-5764556cf6-rzlqd" Jul 29 04:49:54.539613 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.539525 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjfdc\" (UniqueName: \"kubernetes.io/projected/27a57217-5de7-4fd5-8a80-81e13f891264-kube-api-access-cjfdc\") pod \"payload-processing-e2e-api-a-09567c-68bcf8fd5b-btxlb\" (UID: \"27a57217-5de7-4fd5-8a80-81e13f891264\") " pod="openshift-ingress/payload-processing-e2e-api-a-09567c-68bcf8fd5b-btxlb" Jul 29 04:49:54.640515 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.640476 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-09567c-5764556cf6-rzlqd\" (UID: \"8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-09567c-5764556cf6-rzlqd" Jul 29 04:49:54.640725 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.640537 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x78v\" (UniqueName: \"kubernetes.io/projected/8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2-kube-api-access-8x78v\") pod \"payload-pre-processing-e2e-api-a-09567c-5764556cf6-rzlqd\" (UID: \"8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-09567c-5764556cf6-rzlqd" Jul 29 04:49:54.640725 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.640557 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjfdc\" (UniqueName: \"kubernetes.io/projected/27a57217-5de7-4fd5-8a80-81e13f891264-kube-api-access-cjfdc\") pod \"payload-processing-e2e-api-a-09567c-68bcf8fd5b-btxlb\" (UID: \"27a57217-5de7-4fd5-8a80-81e13f891264\") " pod="openshift-ingress/payload-processing-e2e-api-a-09567c-68bcf8fd5b-btxlb" Jul 29 04:49:54.640725 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.640589 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27a57217-5de7-4fd5-8a80-81e13f891264-plugins-config-volume\") pod \"payload-processing-e2e-api-a-09567c-68bcf8fd5b-btxlb\" (UID: \"27a57217-5de7-4fd5-8a80-81e13f891264\") " pod="openshift-ingress/payload-processing-e2e-api-a-09567c-68bcf8fd5b-btxlb" Jul 29 04:49:54.641267 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.641248 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27a57217-5de7-4fd5-8a80-81e13f891264-plugins-config-volume\") pod \"payload-processing-e2e-api-a-09567c-68bcf8fd5b-btxlb\" (UID: \"27a57217-5de7-4fd5-8a80-81e13f891264\") " pod="openshift-ingress/payload-processing-e2e-api-a-09567c-68bcf8fd5b-btxlb" Jul 29 04:49:54.641334 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.641255 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-09567c-5764556cf6-rzlqd\" (UID: \"8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-09567c-5764556cf6-rzlqd" Jul 29 04:49:54.648074 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.648043 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjfdc\" (UniqueName: \"kubernetes.io/projected/27a57217-5de7-4fd5-8a80-81e13f891264-kube-api-access-cjfdc\") pod \"payload-processing-e2e-api-a-09567c-68bcf8fd5b-btxlb\" (UID: \"27a57217-5de7-4fd5-8a80-81e13f891264\") " pod="openshift-ingress/payload-processing-e2e-api-a-09567c-68bcf8fd5b-btxlb" Jul 29 04:49:54.648383 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.648367 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x78v\" (UniqueName: \"kubernetes.io/projected/8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2-kube-api-access-8x78v\") pod \"payload-pre-processing-e2e-api-a-09567c-5764556cf6-rzlqd\" (UID: \"8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-09567c-5764556cf6-rzlqd" Jul 29 04:49:54.714688 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.714654 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-09567c-5764556cf6-rzlqd" Jul 29 04:49:54.753800 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.753767 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-09567c-68bcf8fd5b-btxlb" Jul 29 04:49:54.864007 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.863928 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-09567c-5764556cf6-rzlqd"] Jul 29 04:49:54.909663 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:54.909640 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-09567c-68bcf8fd5b-btxlb"] Jul 29 04:49:54.913286 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:49:54.913261 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a57217_5de7_4fd5_8a80_81e13f891264.slice/crio-744d3793164daf3b53268d118b525a65dd342567a2d4c9adfc413ed50a694d40 WatchSource:0}: Error finding container 744d3793164daf3b53268d118b525a65dd342567a2d4c9adfc413ed50a694d40: Status 404 returned error can't find the container with id 744d3793164daf3b53268d118b525a65dd342567a2d4c9adfc413ed50a694d40 Jul 29 04:49:55.741479 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:55.741436 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-09567c-68bcf8fd5b-btxlb" event={"ID":"27a57217-5de7-4fd5-8a80-81e13f891264","Type":"ContainerStarted","Data":"9d0bd40d0cda2b116ba91465639607062bff3e6b8919d42dd090a156a0534f5a"} Jul 29 04:49:55.741963 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:55.741507 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-09567c-68bcf8fd5b-btxlb" Jul 29 04:49:55.741963 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:55.741535 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-09567c-68bcf8fd5b-btxlb" event={"ID":"27a57217-5de7-4fd5-8a80-81e13f891264","Type":"ContainerStarted","Data":"744d3793164daf3b53268d118b525a65dd342567a2d4c9adfc413ed50a694d40"} Jul 29 04:49:55.743237 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:55.743189 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-09567c-5764556cf6-rzlqd" event={"ID":"8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2","Type":"ContainerStarted","Data":"61ff3685501cfdd30ebaed6924bfb3e2a2e9da457dba4163c09f1b8fdaccb2d6"} Jul 29 04:49:55.743455 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:55.743431 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-09567c-5764556cf6-rzlqd" Jul 29 04:49:55.743563 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:55.743467 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-09567c-5764556cf6-rzlqd" event={"ID":"8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2","Type":"ContainerStarted","Data":"1ce863ef18e7fe22b516cff31993fcac4129ea7625c6f6a226ddf0f58160a263"} Jul 29 04:49:55.756896 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:55.756851 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-09567c-68bcf8fd5b-btxlb" podStartSLOduration=1.756836321 podStartE2EDuration="1.756836321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:49:55.75511958 +0000 UTC m=+2776.614494305" watchObservedRunningTime="2026-07-29 04:49:55.756836321 +0000 UTC m=+2776.616211045" Jul 29 04:49:55.772860 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:49:55.772818 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-09567c-5764556cf6-rzlqd" podStartSLOduration=1.772804638 podStartE2EDuration="1.772804638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:49:55.771846355 +0000 UTC m=+2776.631221079" watchObservedRunningTime="2026-07-29 04:49:55.772804638 +0000 UTC m=+2776.632179361" Jul 29 04:50:06.750926 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:06.750892 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-09567c-68bcf8fd5b-btxlb" Jul 29 04:50:06.751412 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:06.751059 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-09567c-5764556cf6-rzlqd" Jul 29 04:50:22.654256 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:22.654217 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c3ff84-788db79f76-nr9pl"] Jul 29 04:50:22.657747 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:22.657725 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3ff84-788db79f76-nr9pl" Jul 29 04:50:22.660409 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:22.660389 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c3ff84\"" Jul 29 04:50:22.660532 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:22.660497 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c3ff84-dockercfg-skj9q\"" Jul 29 04:50:22.666926 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:22.666905 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c3ff84-788db79f76-nr9pl"] Jul 29 04:50:22.684774 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:22.684735 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c3ff84-8d795c84b-pk6k5"] Jul 29 04:50:22.685103 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:22.685079 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c09d36b5-69c7-4de4-9c88-45cf462f1059-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c3ff84-788db79f76-nr9pl\" (UID: \"c09d36b5-69c7-4de4-9c88-45cf462f1059\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3ff84-788db79f76-nr9pl" Jul 29 04:50:22.685165 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:22.685130 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m5wq\" (UniqueName: \"kubernetes.io/projected/c09d36b5-69c7-4de4-9c88-45cf462f1059-kube-api-access-5m5wq\") pod \"payload-pre-processing-e2e-api-b-c3ff84-788db79f76-nr9pl\" (UID: \"c09d36b5-69c7-4de4-9c88-45cf462f1059\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3ff84-788db79f76-nr9pl" Jul 29 04:50:22.688516 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:22.688495 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c3ff84-8d795c84b-pk6k5" Jul 29 04:50:22.691757 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:22.691734 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c3ff84-8d795c84b-pk6k5"] Jul 29 04:50:22.785735 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:22.785694 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgjnr\" (UniqueName: \"kubernetes.io/projected/87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa-kube-api-access-lgjnr\") pod \"payload-processing-e2e-api-b-c3ff84-8d795c84b-pk6k5\" (UID: \"87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa\") " pod="openshift-ingress/payload-processing-e2e-api-b-c3ff84-8d795c84b-pk6k5" Jul 29 04:50:22.785941 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:22.785767 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c09d36b5-69c7-4de4-9c88-45cf462f1059-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c3ff84-788db79f76-nr9pl\" (UID: \"c09d36b5-69c7-4de4-9c88-45cf462f1059\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3ff84-788db79f76-nr9pl" Jul 29 04:50:22.785941 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:22.785856 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m5wq\" (UniqueName: \"kubernetes.io/projected/c09d36b5-69c7-4de4-9c88-45cf462f1059-kube-api-access-5m5wq\") pod \"payload-pre-processing-e2e-api-b-c3ff84-788db79f76-nr9pl\" (UID: \"c09d36b5-69c7-4de4-9c88-45cf462f1059\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3ff84-788db79f76-nr9pl" Jul 29 04:50:22.785941 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:22.785893 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c3ff84-8d795c84b-pk6k5\" (UID: \"87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa\") " pod="openshift-ingress/payload-processing-e2e-api-b-c3ff84-8d795c84b-pk6k5" Jul 29 04:50:22.786664 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:22.786637 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c09d36b5-69c7-4de4-9c88-45cf462f1059-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c3ff84-788db79f76-nr9pl\" (UID: \"c09d36b5-69c7-4de4-9c88-45cf462f1059\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3ff84-788db79f76-nr9pl" Jul 29 04:50:22.795412 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:22.795389 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m5wq\" (UniqueName: \"kubernetes.io/projected/c09d36b5-69c7-4de4-9c88-45cf462f1059-kube-api-access-5m5wq\") pod \"payload-pre-processing-e2e-api-b-c3ff84-788db79f76-nr9pl\" (UID: \"c09d36b5-69c7-4de4-9c88-45cf462f1059\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3ff84-788db79f76-nr9pl" Jul 29 04:50:22.887188 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:22.887148 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c3ff84-8d795c84b-pk6k5\" (UID: \"87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa\") " pod="openshift-ingress/payload-processing-e2e-api-b-c3ff84-8d795c84b-pk6k5" Jul 29 04:50:22.887407 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:22.887299 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgjnr\" (UniqueName: \"kubernetes.io/projected/87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa-kube-api-access-lgjnr\") pod \"payload-processing-e2e-api-b-c3ff84-8d795c84b-pk6k5\" (UID: \"87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa\") " pod="openshift-ingress/payload-processing-e2e-api-b-c3ff84-8d795c84b-pk6k5" Jul 29 04:50:22.887903 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:22.887879 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c3ff84-8d795c84b-pk6k5\" (UID: \"87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa\") " pod="openshift-ingress/payload-processing-e2e-api-b-c3ff84-8d795c84b-pk6k5" Jul 29 04:50:22.895751 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:22.895722 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgjnr\" (UniqueName: \"kubernetes.io/projected/87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa-kube-api-access-lgjnr\") pod \"payload-processing-e2e-api-b-c3ff84-8d795c84b-pk6k5\" (UID: \"87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa\") " pod="openshift-ingress/payload-processing-e2e-api-b-c3ff84-8d795c84b-pk6k5" Jul 29 04:50:22.970327 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:22.970294 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3ff84-788db79f76-nr9pl" Jul 29 04:50:23.002045 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:23.002014 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c3ff84-8d795c84b-pk6k5" Jul 29 04:50:23.136833 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:23.136772 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c3ff84-788db79f76-nr9pl"] Jul 29 04:50:23.375932 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:23.375717 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c3ff84-8d795c84b-pk6k5"] Jul 29 04:50:23.378114 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:50:23.378074 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87d4b5f7_2d5a_4770_8cf6_69caf59cb4fa.slice/crio-7979ae17cfdff7e597754085910b0090de4d35a77ab4c9622abfc91a7afaed17 WatchSource:0}: Error finding container 7979ae17cfdff7e597754085910b0090de4d35a77ab4c9622abfc91a7afaed17: Status 404 returned error can't find the container with id 7979ae17cfdff7e597754085910b0090de4d35a77ab4c9622abfc91a7afaed17 Jul 29 04:50:23.886669 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:23.886637 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c3ff84-8d795c84b-pk6k5" event={"ID":"87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa","Type":"ContainerStarted","Data":"6233eeac6ca64932ec9fdff85fa1df44366063526abc22d4579c4fdcd5abd0fc"} Jul 29 04:50:23.887176 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:23.886706 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-c3ff84-8d795c84b-pk6k5" Jul 29 04:50:23.887176 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:23.886727 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c3ff84-8d795c84b-pk6k5" event={"ID":"87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa","Type":"ContainerStarted","Data":"7979ae17cfdff7e597754085910b0090de4d35a77ab4c9622abfc91a7afaed17"} Jul 29 04:50:23.888608 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:23.888582 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3ff84-788db79f76-nr9pl" event={"ID":"c09d36b5-69c7-4de4-9c88-45cf462f1059","Type":"ContainerStarted","Data":"3c353fb624a84dc5723435f42c9c0f17c2435b49f601e6749e223390079ec54a"} Jul 29 04:50:23.888794 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:23.888630 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3ff84-788db79f76-nr9pl" event={"ID":"c09d36b5-69c7-4de4-9c88-45cf462f1059","Type":"ContainerStarted","Data":"ad060efba09f32a5f0691404babecabbf4ce16970e865fc92f727300f728c5df"} Jul 29 04:50:23.888794 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:23.888712 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3ff84-788db79f76-nr9pl" Jul 29 04:50:23.902509 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:23.902460 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-c3ff84-8d795c84b-pk6k5" podStartSLOduration=1.902443711 podStartE2EDuration="1.902443711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:50:23.90115079 +0000 UTC m=+2804.760525513" watchObservedRunningTime="2026-07-29 04:50:23.902443711 +0000 UTC m=+2804.761818439" Jul 29 04:50:23.917304 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:23.917258 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3ff84-788db79f76-nr9pl" podStartSLOduration=1.917242917 podStartE2EDuration="1.917242917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:50:23.915497176 +0000 UTC m=+2804.774871902" watchObservedRunningTime="2026-07-29 04:50:23.917242917 +0000 UTC m=+2804.776617641" Jul 29 04:50:34.895240 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:34.895192 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3ff84-788db79f76-nr9pl" Jul 29 04:50:34.895723 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:50:34.895335 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-c3ff84-8d795c84b-pk6k5" Jul 29 04:51:04.940114 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:04.940074 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-09567c-68bcf8fd5b-btxlb"] Jul 29 04:51:04.940614 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:04.940409 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-09567c-68bcf8fd5b-btxlb" podUID="27a57217-5de7-4fd5-8a80-81e13f891264" containerName="payload-processing" containerID="cri-o://9d0bd40d0cda2b116ba91465639607062bff3e6b8919d42dd090a156a0534f5a" gracePeriod=30 Jul 29 04:51:04.957642 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:04.957610 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-09567c-5764556cf6-rzlqd"] Jul 29 04:51:04.958072 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:04.958019 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-09567c-5764556cf6-rzlqd" podUID="8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2" containerName="payload-pre-processing" containerID="cri-o://61ff3685501cfdd30ebaed6924bfb3e2a2e9da457dba4163c09f1b8fdaccb2d6" gracePeriod=30 Jul 29 04:51:05.106375 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:05.106314 2579 generic.go:356] "Generic (PLEG): container finished" podID="27a57217-5de7-4fd5-8a80-81e13f891264" containerID="9d0bd40d0cda2b116ba91465639607062bff3e6b8919d42dd090a156a0534f5a" exitCode=0 Jul 29 04:51:05.106690 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:05.106388 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-09567c-68bcf8fd5b-btxlb" event={"ID":"27a57217-5de7-4fd5-8a80-81e13f891264","Type":"ContainerDied","Data":"9d0bd40d0cda2b116ba91465639607062bff3e6b8919d42dd090a156a0534f5a"} Jul 29 04:51:05.108483 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:05.108456 2579 generic.go:356] "Generic (PLEG): container finished" podID="8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2" containerID="61ff3685501cfdd30ebaed6924bfb3e2a2e9da457dba4163c09f1b8fdaccb2d6" exitCode=0 Jul 29 04:51:05.108598 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:05.108492 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-09567c-5764556cf6-rzlqd" event={"ID":"8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2","Type":"ContainerDied","Data":"61ff3685501cfdd30ebaed6924bfb3e2a2e9da457dba4163c09f1b8fdaccb2d6"} Jul 29 04:51:05.241381 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:05.241356 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-09567c-68bcf8fd5b-btxlb" Jul 29 04:51:05.244316 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:05.244293 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-09567c-5764556cf6-rzlqd" Jul 29 04:51:05.377840 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:05.377804 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjfdc\" (UniqueName: \"kubernetes.io/projected/27a57217-5de7-4fd5-8a80-81e13f891264-kube-api-access-cjfdc\") pod \"27a57217-5de7-4fd5-8a80-81e13f891264\" (UID: \"27a57217-5de7-4fd5-8a80-81e13f891264\") " Jul 29 04:51:05.377840 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:05.377839 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8x78v\" (UniqueName: \"kubernetes.io/projected/8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2-kube-api-access-8x78v\") pod \"8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2\" (UID: \"8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2\") " Jul 29 04:51:05.378068 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:05.377890 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2-plugins-config-volume\") pod \"8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2\" (UID: \"8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2\") " Jul 29 04:51:05.378068 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:05.377946 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27a57217-5de7-4fd5-8a80-81e13f891264-plugins-config-volume\") pod \"27a57217-5de7-4fd5-8a80-81e13f891264\" (UID: \"27a57217-5de7-4fd5-8a80-81e13f891264\") " Jul 29 04:51:05.378363 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:05.378328 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2" (UID: "8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 04:51:05.378489 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:05.378374 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27a57217-5de7-4fd5-8a80-81e13f891264-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "27a57217-5de7-4fd5-8a80-81e13f891264" (UID: "27a57217-5de7-4fd5-8a80-81e13f891264"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 04:51:05.379970 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:05.379948 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27a57217-5de7-4fd5-8a80-81e13f891264-kube-api-access-cjfdc" (OuterVolumeSpecName: "kube-api-access-cjfdc") pod "27a57217-5de7-4fd5-8a80-81e13f891264" (UID: "27a57217-5de7-4fd5-8a80-81e13f891264"). InnerVolumeSpecName "kube-api-access-cjfdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:51:05.380037 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:05.380020 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2-kube-api-access-8x78v" (OuterVolumeSpecName: "kube-api-access-8x78v") pod "8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2" (UID: "8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2"). InnerVolumeSpecName "kube-api-access-8x78v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:51:05.479165 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:05.479072 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2-plugins-config-volume\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:51:05.479165 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:05.479105 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27a57217-5de7-4fd5-8a80-81e13f891264-plugins-config-volume\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:51:05.479165 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:05.479115 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjfdc\" (UniqueName: \"kubernetes.io/projected/27a57217-5de7-4fd5-8a80-81e13f891264-kube-api-access-cjfdc\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:51:05.479165 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:05.479124 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x78v\" (UniqueName: \"kubernetes.io/projected/8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2-kube-api-access-8x78v\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:51:06.114942 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:06.114910 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-09567c-68bcf8fd5b-btxlb" Jul 29 04:51:06.115492 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:06.114955 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-09567c-68bcf8fd5b-btxlb" event={"ID":"27a57217-5de7-4fd5-8a80-81e13f891264","Type":"ContainerDied","Data":"744d3793164daf3b53268d118b525a65dd342567a2d4c9adfc413ed50a694d40"} Jul 29 04:51:06.115492 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:06.115023 2579 scope.go:117] "RemoveContainer" containerID="9d0bd40d0cda2b116ba91465639607062bff3e6b8919d42dd090a156a0534f5a" Jul 29 04:51:06.116397 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:06.116379 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-09567c-5764556cf6-rzlqd" event={"ID":"8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2","Type":"ContainerDied","Data":"1ce863ef18e7fe22b516cff31993fcac4129ea7625c6f6a226ddf0f58160a263"} Jul 29 04:51:06.116465 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:06.116386 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-09567c-5764556cf6-rzlqd" Jul 29 04:51:06.123624 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:06.123604 2579 scope.go:117] "RemoveContainer" containerID="61ff3685501cfdd30ebaed6924bfb3e2a2e9da457dba4163c09f1b8fdaccb2d6" Jul 29 04:51:06.136126 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:06.136102 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-09567c-5764556cf6-rzlqd"] Jul 29 04:51:06.138934 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:06.138908 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-09567c-5764556cf6-rzlqd"] Jul 29 04:51:06.145572 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:06.145549 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-09567c-68bcf8fd5b-btxlb"] Jul 29 04:51:06.149287 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:06.149265 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-09567c-68bcf8fd5b-btxlb"] Jul 29 04:51:07.592522 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:07.592491 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27a57217-5de7-4fd5-8a80-81e13f891264" path="/var/lib/kubelet/pods/27a57217-5de7-4fd5-8a80-81e13f891264/volumes" Jul 29 04:51:07.592921 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:07.592849 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2" path="/var/lib/kubelet/pods/8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2/volumes" Jul 29 04:51:27.314770 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:27.314732 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c3ff84-8d795c84b-pk6k5"] Jul 29 04:51:27.315266 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:27.314977 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-c3ff84-8d795c84b-pk6k5" podUID="87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa" containerName="payload-processing" containerID="cri-o://6233eeac6ca64932ec9fdff85fa1df44366063526abc22d4579c4fdcd5abd0fc" gracePeriod=30 Jul 29 04:51:27.320847 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:27.320819 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c3ff84-788db79f76-nr9pl"] Jul 29 04:51:27.321078 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:27.321048 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3ff84-788db79f76-nr9pl" podUID="c09d36b5-69c7-4de4-9c88-45cf462f1059" containerName="payload-pre-processing" containerID="cri-o://3c353fb624a84dc5723435f42c9c0f17c2435b49f601e6749e223390079ec54a" gracePeriod=30 Jul 29 04:51:27.578158 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:27.578086 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c3ff84-8d795c84b-pk6k5" Jul 29 04:51:27.608757 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:27.608733 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3ff84-788db79f76-nr9pl" Jul 29 04:51:27.675304 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:27.675270 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c09d36b5-69c7-4de4-9c88-45cf462f1059-plugins-config-volume\") pod \"c09d36b5-69c7-4de4-9c88-45cf462f1059\" (UID: \"c09d36b5-69c7-4de4-9c88-45cf462f1059\") " Jul 29 04:51:27.675471 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:27.675317 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgjnr\" (UniqueName: \"kubernetes.io/projected/87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa-kube-api-access-lgjnr\") pod \"87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa\" (UID: \"87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa\") " Jul 29 04:51:27.675471 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:27.675390 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5m5wq\" (UniqueName: \"kubernetes.io/projected/c09d36b5-69c7-4de4-9c88-45cf462f1059-kube-api-access-5m5wq\") pod \"c09d36b5-69c7-4de4-9c88-45cf462f1059\" (UID: \"c09d36b5-69c7-4de4-9c88-45cf462f1059\") " Jul 29 04:51:27.675471 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:27.675427 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa-plugins-config-volume\") pod \"87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa\" (UID: \"87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa\") " Jul 29 04:51:27.675848 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:27.675822 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c09d36b5-69c7-4de4-9c88-45cf462f1059-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c09d36b5-69c7-4de4-9c88-45cf462f1059" (UID: "c09d36b5-69c7-4de4-9c88-45cf462f1059"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 04:51:27.675848 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:27.675835 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa" (UID: "87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 04:51:27.677471 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:27.677448 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa-kube-api-access-lgjnr" (OuterVolumeSpecName: "kube-api-access-lgjnr") pod "87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa" (UID: "87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa"). InnerVolumeSpecName "kube-api-access-lgjnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:51:27.677471 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:27.677466 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c09d36b5-69c7-4de4-9c88-45cf462f1059-kube-api-access-5m5wq" (OuterVolumeSpecName: "kube-api-access-5m5wq") pod "c09d36b5-69c7-4de4-9c88-45cf462f1059" (UID: "c09d36b5-69c7-4de4-9c88-45cf462f1059"). InnerVolumeSpecName "kube-api-access-5m5wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 04:51:27.776943 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:27.776911 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m5wq\" (UniqueName: \"kubernetes.io/projected/c09d36b5-69c7-4de4-9c88-45cf462f1059-kube-api-access-5m5wq\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:51:27.776943 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:27.776936 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa-plugins-config-volume\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:51:27.776943 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:27.776946 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c09d36b5-69c7-4de4-9c88-45cf462f1059-plugins-config-volume\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:51:27.776943 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:27.776955 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgjnr\" (UniqueName: \"kubernetes.io/projected/87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa-kube-api-access-lgjnr\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 04:51:28.234690 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:28.234650 2579 generic.go:356] "Generic (PLEG): container finished" podID="87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa" containerID="6233eeac6ca64932ec9fdff85fa1df44366063526abc22d4579c4fdcd5abd0fc" exitCode=0 Jul 29 04:51:28.234870 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:28.234696 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c3ff84-8d795c84b-pk6k5" Jul 29 04:51:28.234870 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:28.234733 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c3ff84-8d795c84b-pk6k5" event={"ID":"87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa","Type":"ContainerDied","Data":"6233eeac6ca64932ec9fdff85fa1df44366063526abc22d4579c4fdcd5abd0fc"} Jul 29 04:51:28.234870 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:28.234795 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c3ff84-8d795c84b-pk6k5" event={"ID":"87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa","Type":"ContainerDied","Data":"7979ae17cfdff7e597754085910b0090de4d35a77ab4c9622abfc91a7afaed17"} Jul 29 04:51:28.234870 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:28.234823 2579 scope.go:117] "RemoveContainer" containerID="6233eeac6ca64932ec9fdff85fa1df44366063526abc22d4579c4fdcd5abd0fc" Jul 29 04:51:28.236052 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:28.236034 2579 generic.go:356] "Generic (PLEG): container finished" podID="c09d36b5-69c7-4de4-9c88-45cf462f1059" containerID="3c353fb624a84dc5723435f42c9c0f17c2435b49f601e6749e223390079ec54a" exitCode=0 Jul 29 04:51:28.236140 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:28.236079 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3ff84-788db79f76-nr9pl" Jul 29 04:51:28.236140 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:28.236117 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3ff84-788db79f76-nr9pl" event={"ID":"c09d36b5-69c7-4de4-9c88-45cf462f1059","Type":"ContainerDied","Data":"3c353fb624a84dc5723435f42c9c0f17c2435b49f601e6749e223390079ec54a"} Jul 29 04:51:28.236249 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:28.236157 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c3ff84-788db79f76-nr9pl" event={"ID":"c09d36b5-69c7-4de4-9c88-45cf462f1059","Type":"ContainerDied","Data":"ad060efba09f32a5f0691404babecabbf4ce16970e865fc92f727300f728c5df"} Jul 29 04:51:28.243843 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:28.243710 2579 scope.go:117] "RemoveContainer" containerID="6233eeac6ca64932ec9fdff85fa1df44366063526abc22d4579c4fdcd5abd0fc" Jul 29 04:51:28.244057 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:51:28.244025 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6233eeac6ca64932ec9fdff85fa1df44366063526abc22d4579c4fdcd5abd0fc\": container with ID starting with 6233eeac6ca64932ec9fdff85fa1df44366063526abc22d4579c4fdcd5abd0fc not found: ID does not exist" containerID="6233eeac6ca64932ec9fdff85fa1df44366063526abc22d4579c4fdcd5abd0fc" Jul 29 04:51:28.244163 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:28.244066 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6233eeac6ca64932ec9fdff85fa1df44366063526abc22d4579c4fdcd5abd0fc"} err="failed to get container status \"6233eeac6ca64932ec9fdff85fa1df44366063526abc22d4579c4fdcd5abd0fc\": rpc error: code = NotFound desc = could not find container \"6233eeac6ca64932ec9fdff85fa1df44366063526abc22d4579c4fdcd5abd0fc\": container with ID starting with 6233eeac6ca64932ec9fdff85fa1df44366063526abc22d4579c4fdcd5abd0fc not found: ID does not exist" Jul 29 04:51:28.244163 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:28.244091 2579 scope.go:117] "RemoveContainer" containerID="3c353fb624a84dc5723435f42c9c0f17c2435b49f601e6749e223390079ec54a" Jul 29 04:51:28.252083 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:28.252068 2579 scope.go:117] "RemoveContainer" containerID="3c353fb624a84dc5723435f42c9c0f17c2435b49f601e6749e223390079ec54a" Jul 29 04:51:28.252336 ip-10-0-139-111 kubenswrapper[2579]: E0729 04:51:28.252312 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c353fb624a84dc5723435f42c9c0f17c2435b49f601e6749e223390079ec54a\": container with ID starting with 3c353fb624a84dc5723435f42c9c0f17c2435b49f601e6749e223390079ec54a not found: ID does not exist" containerID="3c353fb624a84dc5723435f42c9c0f17c2435b49f601e6749e223390079ec54a" Jul 29 04:51:28.252419 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:28.252339 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c353fb624a84dc5723435f42c9c0f17c2435b49f601e6749e223390079ec54a"} err="failed to get container status \"3c353fb624a84dc5723435f42c9c0f17c2435b49f601e6749e223390079ec54a\": rpc error: code = NotFound desc = could not find container \"3c353fb624a84dc5723435f42c9c0f17c2435b49f601e6749e223390079ec54a\": container with ID starting with 3c353fb624a84dc5723435f42c9c0f17c2435b49f601e6749e223390079ec54a not found: ID does not exist" Jul 29 04:51:28.263456 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:28.263435 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c3ff84-788db79f76-nr9pl"] Jul 29 04:51:28.270930 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:28.270910 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c3ff84-788db79f76-nr9pl"] Jul 29 04:51:28.283485 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:28.283461 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c3ff84-8d795c84b-pk6k5"] Jul 29 04:51:28.289341 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:28.289312 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c3ff84-8d795c84b-pk6k5"] Jul 29 04:51:29.591368 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:29.591336 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa" path="/var/lib/kubelet/pods/87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa/volumes" Jul 29 04:51:29.591754 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:29.591694 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c09d36b5-69c7-4de4-9c88-45cf462f1059" path="/var/lib/kubelet/pods/c09d36b5-69c7-4de4-9c88-45cf462f1059/volumes" Jul 29 04:51:39.045999 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.045965 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-6eef04-8469cf6cb8-5vxlk"] Jul 29 04:51:39.046528 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.046362 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2" containerName="payload-pre-processing" Jul 29 04:51:39.046528 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.046376 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2" containerName="payload-pre-processing" Jul 29 04:51:39.046528 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.046385 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c09d36b5-69c7-4de4-9c88-45cf462f1059" containerName="payload-pre-processing" Jul 29 04:51:39.046528 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.046391 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c09d36b5-69c7-4de4-9c88-45cf462f1059" containerName="payload-pre-processing" Jul 29 04:51:39.046528 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.046397 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa" containerName="payload-processing" Jul 29 04:51:39.046528 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.046403 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa" containerName="payload-processing" Jul 29 04:51:39.046528 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.046419 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27a57217-5de7-4fd5-8a80-81e13f891264" containerName="payload-processing" Jul 29 04:51:39.046528 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.046424 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="27a57217-5de7-4fd5-8a80-81e13f891264" containerName="payload-processing" Jul 29 04:51:39.046528 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.046476 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="87d4b5f7-2d5a-4770-8cf6-69caf59cb4fa" containerName="payload-processing" Jul 29 04:51:39.046528 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.046483 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c09d36b5-69c7-4de4-9c88-45cf462f1059" containerName="payload-pre-processing" Jul 29 04:51:39.046528 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.046491 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fb8d2ad-36ee-4ddc-a4ae-6cae5b64e3c2" containerName="payload-pre-processing" Jul 29 04:51:39.046528 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.046499 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="27a57217-5de7-4fd5-8a80-81e13f891264" containerName="payload-processing" Jul 29 04:51:39.050794 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.050777 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6eef04-8469cf6cb8-5vxlk" Jul 29 04:51:39.053487 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.053464 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-6eef04\"" Jul 29 04:51:39.053630 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.053579 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-6eef04-dockercfg-w8qxr\"" Jul 29 04:51:39.060139 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.060073 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-6eef04-8469cf6cb8-5vxlk"] Jul 29 04:51:39.062642 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.062623 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-6eef04-76c4cb4c9d-4l5kj"] Jul 29 04:51:39.065977 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.065958 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-6eef04-76c4cb4c9d-4l5kj" Jul 29 04:51:39.076127 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.076102 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-6eef04-76c4cb4c9d-4l5kj"] Jul 29 04:51:39.179271 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.179236 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/453ac86d-ebcd-47d0-97a4-5a94969abffa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-6eef04-8469cf6cb8-5vxlk\" (UID: \"453ac86d-ebcd-47d0-97a4-5a94969abffa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6eef04-8469cf6cb8-5vxlk" Jul 29 04:51:39.179271 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.179276 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b6103e8-6946-431b-b34d-5d6b63739df7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-6eef04-76c4cb4c9d-4l5kj\" (UID: \"6b6103e8-6946-431b-b34d-5d6b63739df7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-6eef04-76c4cb4c9d-4l5kj" Jul 29 04:51:39.179497 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.179297 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqtwz\" (UniqueName: \"kubernetes.io/projected/453ac86d-ebcd-47d0-97a4-5a94969abffa-kube-api-access-gqtwz\") pod \"payload-pre-processing-e2e-shared-a-6eef04-8469cf6cb8-5vxlk\" (UID: \"453ac86d-ebcd-47d0-97a4-5a94969abffa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6eef04-8469cf6cb8-5vxlk" Jul 29 04:51:39.179497 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.179344 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pjtx\" (UniqueName: \"kubernetes.io/projected/6b6103e8-6946-431b-b34d-5d6b63739df7-kube-api-access-9pjtx\") pod \"payload-processing-e2e-shared-a-6eef04-76c4cb4c9d-4l5kj\" (UID: \"6b6103e8-6946-431b-b34d-5d6b63739df7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-6eef04-76c4cb4c9d-4l5kj" Jul 29 04:51:39.280504 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.280467 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pjtx\" (UniqueName: \"kubernetes.io/projected/6b6103e8-6946-431b-b34d-5d6b63739df7-kube-api-access-9pjtx\") pod \"payload-processing-e2e-shared-a-6eef04-76c4cb4c9d-4l5kj\" (UID: \"6b6103e8-6946-431b-b34d-5d6b63739df7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-6eef04-76c4cb4c9d-4l5kj" Jul 29 04:51:39.280703 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.280548 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/453ac86d-ebcd-47d0-97a4-5a94969abffa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-6eef04-8469cf6cb8-5vxlk\" (UID: \"453ac86d-ebcd-47d0-97a4-5a94969abffa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6eef04-8469cf6cb8-5vxlk" Jul 29 04:51:39.280703 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.280566 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b6103e8-6946-431b-b34d-5d6b63739df7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-6eef04-76c4cb4c9d-4l5kj\" (UID: \"6b6103e8-6946-431b-b34d-5d6b63739df7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-6eef04-76c4cb4c9d-4l5kj" Jul 29 04:51:39.280703 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.280584 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqtwz\" (UniqueName: \"kubernetes.io/projected/453ac86d-ebcd-47d0-97a4-5a94969abffa-kube-api-access-gqtwz\") pod \"payload-pre-processing-e2e-shared-a-6eef04-8469cf6cb8-5vxlk\" (UID: \"453ac86d-ebcd-47d0-97a4-5a94969abffa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6eef04-8469cf6cb8-5vxlk" Jul 29 04:51:39.281305 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.281282 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b6103e8-6946-431b-b34d-5d6b63739df7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-6eef04-76c4cb4c9d-4l5kj\" (UID: \"6b6103e8-6946-431b-b34d-5d6b63739df7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-6eef04-76c4cb4c9d-4l5kj" Jul 29 04:51:39.281415 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.281338 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/453ac86d-ebcd-47d0-97a4-5a94969abffa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-6eef04-8469cf6cb8-5vxlk\" (UID: \"453ac86d-ebcd-47d0-97a4-5a94969abffa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6eef04-8469cf6cb8-5vxlk" Jul 29 04:51:39.288687 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.288665 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqtwz\" (UniqueName: \"kubernetes.io/projected/453ac86d-ebcd-47d0-97a4-5a94969abffa-kube-api-access-gqtwz\") pod \"payload-pre-processing-e2e-shared-a-6eef04-8469cf6cb8-5vxlk\" (UID: \"453ac86d-ebcd-47d0-97a4-5a94969abffa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6eef04-8469cf6cb8-5vxlk" Jul 29 04:51:39.289042 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.289019 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pjtx\" (UniqueName: \"kubernetes.io/projected/6b6103e8-6946-431b-b34d-5d6b63739df7-kube-api-access-9pjtx\") pod \"payload-processing-e2e-shared-a-6eef04-76c4cb4c9d-4l5kj\" (UID: \"6b6103e8-6946-431b-b34d-5d6b63739df7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-6eef04-76c4cb4c9d-4l5kj" Jul 29 04:51:39.363073 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.362977 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6eef04-8469cf6cb8-5vxlk" Jul 29 04:51:39.378421 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.378394 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-6eef04-76c4cb4c9d-4l5kj" Jul 29 04:51:39.545974 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.545940 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-6eef04-8469cf6cb8-5vxlk"] Jul 29 04:51:39.548389 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:51:39.548357 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453ac86d_ebcd_47d0_97a4_5a94969abffa.slice/crio-4911cc5eb57d3d2a78a1af19229c363727a361904838a69e2355d3753de81af6 WatchSource:0}: Error finding container 4911cc5eb57d3d2a78a1af19229c363727a361904838a69e2355d3753de81af6: Status 404 returned error can't find the container with id 4911cc5eb57d3d2a78a1af19229c363727a361904838a69e2355d3753de81af6 Jul 29 04:51:39.571040 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:39.571014 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-6eef04-76c4cb4c9d-4l5kj"] Jul 29 04:51:39.585183 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:51:39.585140 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6103e8_6946_431b_b34d_5d6b63739df7.slice/crio-edacb7a1db1809c7e2e96e845b21ff97f26a6ebfe6d9e1fc607d2830cccbcff0 WatchSource:0}: Error finding container edacb7a1db1809c7e2e96e845b21ff97f26a6ebfe6d9e1fc607d2830cccbcff0: Status 404 returned error can't find the container with id edacb7a1db1809c7e2e96e845b21ff97f26a6ebfe6d9e1fc607d2830cccbcff0 Jul 29 04:51:40.299125 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:40.299090 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-6eef04-76c4cb4c9d-4l5kj" event={"ID":"6b6103e8-6946-431b-b34d-5d6b63739df7","Type":"ContainerStarted","Data":"90b46f0303ac3c477802fc73659eb18c0f4c8606e908251c27a47f13ab4672b9"} Jul 29 04:51:40.299609 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:40.299147 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-6eef04-76c4cb4c9d-4l5kj" event={"ID":"6b6103e8-6946-431b-b34d-5d6b63739df7","Type":"ContainerStarted","Data":"edacb7a1db1809c7e2e96e845b21ff97f26a6ebfe6d9e1fc607d2830cccbcff0"} Jul 29 04:51:40.299609 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:40.299236 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-6eef04-76c4cb4c9d-4l5kj" Jul 29 04:51:40.300609 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:40.300587 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6eef04-8469cf6cb8-5vxlk" event={"ID":"453ac86d-ebcd-47d0-97a4-5a94969abffa","Type":"ContainerStarted","Data":"7a4a39456779a105841a58ba71fe28e05a7da43056c398b88d6f2f276caa81f4"} Jul 29 04:51:40.300743 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:40.300625 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6eef04-8469cf6cb8-5vxlk" event={"ID":"453ac86d-ebcd-47d0-97a4-5a94969abffa","Type":"ContainerStarted","Data":"4911cc5eb57d3d2a78a1af19229c363727a361904838a69e2355d3753de81af6"} Jul 29 04:51:40.300743 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:40.300684 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6eef04-8469cf6cb8-5vxlk" Jul 29 04:51:40.317226 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:40.317163 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-6eef04-76c4cb4c9d-4l5kj" podStartSLOduration=1.3171483099999999 podStartE2EDuration="1.31714831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:51:40.31613116 +0000 UTC m=+2881.175505885" watchObservedRunningTime="2026-07-29 04:51:40.31714831 +0000 UTC m=+2881.176523033" Jul 29 04:51:40.332140 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:40.332094 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6eef04-8469cf6cb8-5vxlk" podStartSLOduration=1.332080068 podStartE2EDuration="1.332080068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:51:40.330010011 +0000 UTC m=+2881.189384734" watchObservedRunningTime="2026-07-29 04:51:40.332080068 +0000 UTC m=+2881.191454793" Jul 29 04:51:51.307686 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:51.307652 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-6eef04-76c4cb4c9d-4l5kj" Jul 29 04:51:51.308097 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:51:51.307721 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6eef04-8469cf6cb8-5vxlk" Jul 29 04:52:07.132580 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:07.132545 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-be8bc5-57d59449d8-w8h52"] Jul 29 04:52:07.142544 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:07.142519 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-be8bc5-57d59449d8-w8h52" Jul 29 04:52:07.145084 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:07.145061 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-be8bc5-dockercfg-2bzmg\"" Jul 29 04:52:07.145230 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:07.145071 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-be8bc5\"" Jul 29 04:52:07.145431 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:07.145412 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-be8bc5-57d59449d8-w8h52"] Jul 29 04:52:07.175324 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:07.175300 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-be8bc5-6f6c4fd6f8-l882t"] Jul 29 04:52:07.178758 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:07.178739 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-be8bc5-6f6c4fd6f8-l882t" Jul 29 04:52:07.189162 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:07.189122 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-be8bc5-6f6c4fd6f8-l882t"] Jul 29 04:52:07.232011 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:07.231970 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqfhz\" (UniqueName: \"kubernetes.io/projected/90926e7a-4e34-473a-af59-d28100545dc0-kube-api-access-sqfhz\") pod \"payload-pre-processing-e2e-shared-b-be8bc5-57d59449d8-w8h52\" (UID: \"90926e7a-4e34-473a-af59-d28100545dc0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-be8bc5-57d59449d8-w8h52" Jul 29 04:52:07.232239 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:07.232049 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sqmr\" (UniqueName: \"kubernetes.io/projected/353191af-c906-4fa7-b567-23bcb9d54318-kube-api-access-9sqmr\") pod \"payload-processing-e2e-shared-b-be8bc5-6f6c4fd6f8-l882t\" (UID: \"353191af-c906-4fa7-b567-23bcb9d54318\") " pod="openshift-ingress/payload-processing-e2e-shared-b-be8bc5-6f6c4fd6f8-l882t" Jul 29 04:52:07.232239 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:07.232092 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353191af-c906-4fa7-b567-23bcb9d54318-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-be8bc5-6f6c4fd6f8-l882t\" (UID: \"353191af-c906-4fa7-b567-23bcb9d54318\") " pod="openshift-ingress/payload-processing-e2e-shared-b-be8bc5-6f6c4fd6f8-l882t" Jul 29 04:52:07.232239 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:07.232178 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90926e7a-4e34-473a-af59-d28100545dc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-be8bc5-57d59449d8-w8h52\" (UID: \"90926e7a-4e34-473a-af59-d28100545dc0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-be8bc5-57d59449d8-w8h52" Jul 29 04:52:07.333158 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:07.333121 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90926e7a-4e34-473a-af59-d28100545dc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-be8bc5-57d59449d8-w8h52\" (UID: \"90926e7a-4e34-473a-af59-d28100545dc0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-be8bc5-57d59449d8-w8h52" Jul 29 04:52:07.333353 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:07.333195 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqfhz\" (UniqueName: \"kubernetes.io/projected/90926e7a-4e34-473a-af59-d28100545dc0-kube-api-access-sqfhz\") pod \"payload-pre-processing-e2e-shared-b-be8bc5-57d59449d8-w8h52\" (UID: \"90926e7a-4e34-473a-af59-d28100545dc0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-be8bc5-57d59449d8-w8h52" Jul 29 04:52:07.333353 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:07.333256 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sqmr\" (UniqueName: \"kubernetes.io/projected/353191af-c906-4fa7-b567-23bcb9d54318-kube-api-access-9sqmr\") pod \"payload-processing-e2e-shared-b-be8bc5-6f6c4fd6f8-l882t\" (UID: \"353191af-c906-4fa7-b567-23bcb9d54318\") " pod="openshift-ingress/payload-processing-e2e-shared-b-be8bc5-6f6c4fd6f8-l882t" Jul 29 04:52:07.333353 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:07.333291 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353191af-c906-4fa7-b567-23bcb9d54318-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-be8bc5-6f6c4fd6f8-l882t\" (UID: \"353191af-c906-4fa7-b567-23bcb9d54318\") " pod="openshift-ingress/payload-processing-e2e-shared-b-be8bc5-6f6c4fd6f8-l882t" Jul 29 04:52:07.333899 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:07.333872 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90926e7a-4e34-473a-af59-d28100545dc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-be8bc5-57d59449d8-w8h52\" (UID: \"90926e7a-4e34-473a-af59-d28100545dc0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-be8bc5-57d59449d8-w8h52" Jul 29 04:52:07.334115 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:07.334089 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353191af-c906-4fa7-b567-23bcb9d54318-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-be8bc5-6f6c4fd6f8-l882t\" (UID: \"353191af-c906-4fa7-b567-23bcb9d54318\") " pod="openshift-ingress/payload-processing-e2e-shared-b-be8bc5-6f6c4fd6f8-l882t" Jul 29 04:52:07.342117 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:07.342097 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sqmr\" (UniqueName: \"kubernetes.io/projected/353191af-c906-4fa7-b567-23bcb9d54318-kube-api-access-9sqmr\") pod \"payload-processing-e2e-shared-b-be8bc5-6f6c4fd6f8-l882t\" (UID: \"353191af-c906-4fa7-b567-23bcb9d54318\") " pod="openshift-ingress/payload-processing-e2e-shared-b-be8bc5-6f6c4fd6f8-l882t" Jul 29 04:52:07.343419 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:07.343399 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqfhz\" (UniqueName: \"kubernetes.io/projected/90926e7a-4e34-473a-af59-d28100545dc0-kube-api-access-sqfhz\") pod \"payload-pre-processing-e2e-shared-b-be8bc5-57d59449d8-w8h52\" (UID: \"90926e7a-4e34-473a-af59-d28100545dc0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-be8bc5-57d59449d8-w8h52" Jul 29 04:52:07.457347 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:07.457311 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-be8bc5-57d59449d8-w8h52" Jul 29 04:52:07.489598 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:07.489567 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-be8bc5-6f6c4fd6f8-l882t" Jul 29 04:52:07.620634 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:07.620601 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-be8bc5-57d59449d8-w8h52"] Jul 29 04:52:07.622427 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:52:07.622398 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90926e7a_4e34_473a_af59_d28100545dc0.slice/crio-4280c266d0ecb3fd39104c97cbd73e02a4b8f9ea38f50a507c6e910396731a75 WatchSource:0}: Error finding container 4280c266d0ecb3fd39104c97cbd73e02a4b8f9ea38f50a507c6e910396731a75: Status 404 returned error can't find the container with id 4280c266d0ecb3fd39104c97cbd73e02a4b8f9ea38f50a507c6e910396731a75 Jul 29 04:52:07.648986 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:07.648959 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-be8bc5-6f6c4fd6f8-l882t"] Jul 29 04:52:07.653347 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:52:07.653324 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod353191af_c906_4fa7_b567_23bcb9d54318.slice/crio-fa86d468c772ef618132823785ca25240ee6facd260b464f0595124bfd592325 WatchSource:0}: Error finding container fa86d468c772ef618132823785ca25240ee6facd260b464f0595124bfd592325: Status 404 returned error can't find the container with id fa86d468c772ef618132823785ca25240ee6facd260b464f0595124bfd592325 Jul 29 04:52:08.441644 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:08.441606 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-be8bc5-57d59449d8-w8h52" event={"ID":"90926e7a-4e34-473a-af59-d28100545dc0","Type":"ContainerStarted","Data":"c84748e8e6b3d0e79ca0d65205581ef25ffdf675ed4d978818347aa16cd8fd55"} Jul 29 04:52:08.442157 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:08.441673 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-be8bc5-57d59449d8-w8h52" Jul 29 04:52:08.442157 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:08.441698 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-be8bc5-57d59449d8-w8h52" event={"ID":"90926e7a-4e34-473a-af59-d28100545dc0","Type":"ContainerStarted","Data":"4280c266d0ecb3fd39104c97cbd73e02a4b8f9ea38f50a507c6e910396731a75"} Jul 29 04:52:08.443242 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:08.443197 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-be8bc5-6f6c4fd6f8-l882t" event={"ID":"353191af-c906-4fa7-b567-23bcb9d54318","Type":"ContainerStarted","Data":"6f338524a5f34caaa6af410dc9c98ec8d5e186f2530e6bff1bbc431b65294c7f"} Jul 29 04:52:08.443369 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:08.443260 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-be8bc5-6f6c4fd6f8-l882t" event={"ID":"353191af-c906-4fa7-b567-23bcb9d54318","Type":"ContainerStarted","Data":"fa86d468c772ef618132823785ca25240ee6facd260b464f0595124bfd592325"} Jul 29 04:52:08.443369 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:08.443290 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-be8bc5-6f6c4fd6f8-l882t" Jul 29 04:52:08.460862 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:08.460813 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-be8bc5-57d59449d8-w8h52" podStartSLOduration=1.460794936 podStartE2EDuration="1.460794936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:52:08.457708973 +0000 UTC m=+2909.317083696" watchObservedRunningTime="2026-07-29 04:52:08.460794936 +0000 UTC m=+2909.320169663" Jul 29 04:52:08.474396 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:08.474354 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-be8bc5-6f6c4fd6f8-l882t" podStartSLOduration=1.474341168 podStartE2EDuration="1.474341168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:52:08.473191761 +0000 UTC m=+2909.332566487" watchObservedRunningTime="2026-07-29 04:52:08.474341168 +0000 UTC m=+2909.333715933" Jul 29 04:52:19.450190 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:19.450135 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-be8bc5-57d59449d8-w8h52" podUID="90926e7a-4e34-473a-af59-d28100545dc0" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.72:9004: connect: connection refused" Jul 29 04:52:19.450757 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:19.450645 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-be8bc5-6f6c4fd6f8-l882t" Jul 29 04:52:29.449025 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:52:29.448993 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-be8bc5-57d59449d8-w8h52" Jul 29 04:58:22.416386 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:22.416353 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw"] Jul 29 04:58:22.420134 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:22.420111 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" Jul 29 04:58:22.423172 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:22.423140 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-6eef04\"/\"kube-root-ca.crt\"" Jul 29 04:58:22.423330 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:22.423169 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6eef04\"/\"sub-test-model-6eef04-kserve-self-signed-certs\"" Jul 29 04:58:22.423330 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:22.423175 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6eef04\"/\"default-dockercfg-w4mtl\"" Jul 29 04:58:22.423330 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:22.423268 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-6eef04\"/\"openshift-service-ca.crt\"" Jul 29 04:58:22.427364 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:22.427341 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw"] Jul 29 04:58:22.493619 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:22.493573 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e1f6719-f4ec-4c1e-96c3-29de41a22219-dshm\") pod \"sub-test-model-6eef04-kserve-79c5cbcddc-2cczw\" (UID: \"4e1f6719-f4ec-4c1e-96c3-29de41a22219\") " pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" Jul 29 04:58:22.493834 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:22.493631 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e1f6719-f4ec-4c1e-96c3-29de41a22219-model-cache\") pod \"sub-test-model-6eef04-kserve-79c5cbcddc-2cczw\" (UID: \"4e1f6719-f4ec-4c1e-96c3-29de41a22219\") " pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" Jul 29 04:58:22.493834 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:22.493655 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzxq9\" (UniqueName: \"kubernetes.io/projected/4e1f6719-f4ec-4c1e-96c3-29de41a22219-kube-api-access-xzxq9\") pod \"sub-test-model-6eef04-kserve-79c5cbcddc-2cczw\" (UID: \"4e1f6719-f4ec-4c1e-96c3-29de41a22219\") " pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" Jul 29 04:58:22.493834 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:22.493774 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e1f6719-f4ec-4c1e-96c3-29de41a22219-home\") pod \"sub-test-model-6eef04-kserve-79c5cbcddc-2cczw\" (UID: \"4e1f6719-f4ec-4c1e-96c3-29de41a22219\") " pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" Jul 29 04:58:22.493834 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:22.493814 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e1f6719-f4ec-4c1e-96c3-29de41a22219-tls-certs\") pod \"sub-test-model-6eef04-kserve-79c5cbcddc-2cczw\" (UID: \"4e1f6719-f4ec-4c1e-96c3-29de41a22219\") " pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" Jul 29 04:58:22.494016 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:22.493847 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e1f6719-f4ec-4c1e-96c3-29de41a22219-tmp-dir\") pod \"sub-test-model-6eef04-kserve-79c5cbcddc-2cczw\" (UID: \"4e1f6719-f4ec-4c1e-96c3-29de41a22219\") " pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" Jul 29 04:58:22.595136 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:22.595100 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e1f6719-f4ec-4c1e-96c3-29de41a22219-home\") pod \"sub-test-model-6eef04-kserve-79c5cbcddc-2cczw\" (UID: \"4e1f6719-f4ec-4c1e-96c3-29de41a22219\") " pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" Jul 29 04:58:22.595367 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:22.595144 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e1f6719-f4ec-4c1e-96c3-29de41a22219-tls-certs\") pod \"sub-test-model-6eef04-kserve-79c5cbcddc-2cczw\" (UID: \"4e1f6719-f4ec-4c1e-96c3-29de41a22219\") " pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" Jul 29 04:58:22.595367 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:22.595175 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e1f6719-f4ec-4c1e-96c3-29de41a22219-tmp-dir\") pod \"sub-test-model-6eef04-kserve-79c5cbcddc-2cczw\" (UID: \"4e1f6719-f4ec-4c1e-96c3-29de41a22219\") " pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" Jul 29 04:58:22.595367 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:22.595259 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e1f6719-f4ec-4c1e-96c3-29de41a22219-dshm\") pod \"sub-test-model-6eef04-kserve-79c5cbcddc-2cczw\" (UID: \"4e1f6719-f4ec-4c1e-96c3-29de41a22219\") " pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" Jul 29 04:58:22.595367 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:22.595304 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e1f6719-f4ec-4c1e-96c3-29de41a22219-model-cache\") pod \"sub-test-model-6eef04-kserve-79c5cbcddc-2cczw\" (UID: \"4e1f6719-f4ec-4c1e-96c3-29de41a22219\") " pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" Jul 29 04:58:22.595367 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:22.595337 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzxq9\" (UniqueName: \"kubernetes.io/projected/4e1f6719-f4ec-4c1e-96c3-29de41a22219-kube-api-access-xzxq9\") pod \"sub-test-model-6eef04-kserve-79c5cbcddc-2cczw\" (UID: \"4e1f6719-f4ec-4c1e-96c3-29de41a22219\") " pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" Jul 29 04:58:22.595665 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:22.595585 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e1f6719-f4ec-4c1e-96c3-29de41a22219-home\") pod \"sub-test-model-6eef04-kserve-79c5cbcddc-2cczw\" (UID: \"4e1f6719-f4ec-4c1e-96c3-29de41a22219\") " pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" Jul 29 04:58:22.595872 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:22.595846 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e1f6719-f4ec-4c1e-96c3-29de41a22219-model-cache\") pod \"sub-test-model-6eef04-kserve-79c5cbcddc-2cczw\" (UID: \"4e1f6719-f4ec-4c1e-96c3-29de41a22219\") " pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" Jul 29 04:58:22.596062 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:22.596040 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e1f6719-f4ec-4c1e-96c3-29de41a22219-tmp-dir\") pod \"sub-test-model-6eef04-kserve-79c5cbcddc-2cczw\" (UID: \"4e1f6719-f4ec-4c1e-96c3-29de41a22219\") " pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" Jul 29 04:58:22.597915 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:22.597894 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e1f6719-f4ec-4c1e-96c3-29de41a22219-tls-certs\") pod \"sub-test-model-6eef04-kserve-79c5cbcddc-2cczw\" (UID: \"4e1f6719-f4ec-4c1e-96c3-29de41a22219\") " pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" Jul 29 04:58:22.598093 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:22.598075 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e1f6719-f4ec-4c1e-96c3-29de41a22219-dshm\") pod \"sub-test-model-6eef04-kserve-79c5cbcddc-2cczw\" (UID: \"4e1f6719-f4ec-4c1e-96c3-29de41a22219\") " pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" Jul 29 04:58:22.602577 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:22.602537 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzxq9\" (UniqueName: \"kubernetes.io/projected/4e1f6719-f4ec-4c1e-96c3-29de41a22219-kube-api-access-xzxq9\") pod \"sub-test-model-6eef04-kserve-79c5cbcddc-2cczw\" (UID: \"4e1f6719-f4ec-4c1e-96c3-29de41a22219\") " pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" Jul 29 04:58:22.731238 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:22.731171 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" Jul 29 04:58:22.861775 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:22.861745 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw"] Jul 29 04:58:22.863402 ip-10-0-139-111 kubenswrapper[2579]: W0729 04:58:22.863372 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e1f6719_f4ec_4c1e_96c3_29de41a22219.slice/crio-1e38633c7f46484c813f5d8e81fb9f80fd2e6f9cd6d5fb5172ee057b170a96a1 WatchSource:0}: Error finding container 1e38633c7f46484c813f5d8e81fb9f80fd2e6f9cd6d5fb5172ee057b170a96a1: Status 404 returned error can't find the container with id 1e38633c7f46484c813f5d8e81fb9f80fd2e6f9cd6d5fb5172ee057b170a96a1 Jul 29 04:58:23.294155 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:23.294111 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" event={"ID":"4e1f6719-f4ec-4c1e-96c3-29de41a22219","Type":"ContainerStarted","Data":"cbc317a29a735eb01aa4bf2127771ee1ef957d673307bbcfa616a5829c0d897e"} Jul 29 04:58:23.294370 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:23.294165 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" event={"ID":"4e1f6719-f4ec-4c1e-96c3-29de41a22219","Type":"ContainerStarted","Data":"1e38633c7f46484c813f5d8e81fb9f80fd2e6f9cd6d5fb5172ee057b170a96a1"} Jul 29 04:58:23.310411 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:23.310359 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" podStartSLOduration=1.310344232 podStartE2EDuration="1.310344232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 04:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 04:58:23.308663717 +0000 UTC m=+3284.168038537" watchObservedRunningTime="2026-07-29 04:58:23.310344232 +0000 UTC m=+3284.169718956" Jul 29 04:58:32.732007 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:32.731972 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" Jul 29 04:58:32.732518 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:32.732028 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" Jul 29 04:58:32.745296 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:32.745271 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" Jul 29 04:58:33.356155 ip-10-0-139-111 kubenswrapper[2579]: I0729 04:58:33.356126 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" Jul 29 05:00:04.386449 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:04.386415 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754975-d9tln"] Jul 29 05:00:04.391259 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:04.391228 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754975-d9tln"] Jul 29 05:00:05.591727 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:05.591693 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d5d0d12-d04f-4cd0-bf20-bf7cd3b7b688" path="/var/lib/kubelet/pods/7d5d0d12-d04f-4cd0-bf20-bf7cd3b7b688/volumes" Jul 29 05:00:19.710696 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:19.710649 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw"] Jul 29 05:00:34.730842 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:34.730758 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" podUID="4e1f6719-f4ec-4c1e-96c3-29de41a22219" containerName="main" containerID="cri-o://cbc317a29a735eb01aa4bf2127771ee1ef957d673307bbcfa616a5829c0d897e" gracePeriod=45 Jul 29 05:00:34.949399 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:34.949367 2579 generic.go:356] "Generic (PLEG): container finished" podID="4e1f6719-f4ec-4c1e-96c3-29de41a22219" containerID="cbc317a29a735eb01aa4bf2127771ee1ef957d673307bbcfa616a5829c0d897e" exitCode=0 Jul 29 05:00:34.949610 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:34.949415 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" event={"ID":"4e1f6719-f4ec-4c1e-96c3-29de41a22219","Type":"ContainerDied","Data":"cbc317a29a735eb01aa4bf2127771ee1ef957d673307bbcfa616a5829c0d897e"} Jul 29 05:00:34.992098 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:34.992035 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" Jul 29 05:00:35.107181 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:35.107141 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e1f6719-f4ec-4c1e-96c3-29de41a22219-model-cache\") pod \"4e1f6719-f4ec-4c1e-96c3-29de41a22219\" (UID: \"4e1f6719-f4ec-4c1e-96c3-29de41a22219\") " Jul 29 05:00:35.107181 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:35.107187 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e1f6719-f4ec-4c1e-96c3-29de41a22219-dshm\") pod \"4e1f6719-f4ec-4c1e-96c3-29de41a22219\" (UID: \"4e1f6719-f4ec-4c1e-96c3-29de41a22219\") " Jul 29 05:00:35.107519 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:35.107265 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e1f6719-f4ec-4c1e-96c3-29de41a22219-tls-certs\") pod \"4e1f6719-f4ec-4c1e-96c3-29de41a22219\" (UID: \"4e1f6719-f4ec-4c1e-96c3-29de41a22219\") " Jul 29 05:00:35.107519 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:35.107298 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e1f6719-f4ec-4c1e-96c3-29de41a22219-home\") pod \"4e1f6719-f4ec-4c1e-96c3-29de41a22219\" (UID: \"4e1f6719-f4ec-4c1e-96c3-29de41a22219\") " Jul 29 05:00:35.107519 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:35.107331 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzxq9\" (UniqueName: \"kubernetes.io/projected/4e1f6719-f4ec-4c1e-96c3-29de41a22219-kube-api-access-xzxq9\") pod \"4e1f6719-f4ec-4c1e-96c3-29de41a22219\" (UID: \"4e1f6719-f4ec-4c1e-96c3-29de41a22219\") " Jul 29 05:00:35.107519 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:35.107371 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e1f6719-f4ec-4c1e-96c3-29de41a22219-tmp-dir\") pod \"4e1f6719-f4ec-4c1e-96c3-29de41a22219\" (UID: \"4e1f6719-f4ec-4c1e-96c3-29de41a22219\") " Jul 29 05:00:35.107519 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:35.107456 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1f6719-f4ec-4c1e-96c3-29de41a22219-model-cache" (OuterVolumeSpecName: "model-cache") pod "4e1f6719-f4ec-4c1e-96c3-29de41a22219" (UID: "4e1f6719-f4ec-4c1e-96c3-29de41a22219"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 05:00:35.107781 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:35.107622 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1f6719-f4ec-4c1e-96c3-29de41a22219-home" (OuterVolumeSpecName: "home") pod "4e1f6719-f4ec-4c1e-96c3-29de41a22219" (UID: "4e1f6719-f4ec-4c1e-96c3-29de41a22219"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 05:00:35.107781 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:35.107660 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1f6719-f4ec-4c1e-96c3-29de41a22219-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e1f6719-f4ec-4c1e-96c3-29de41a22219" (UID: "4e1f6719-f4ec-4c1e-96c3-29de41a22219"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 05:00:35.107781 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:35.107726 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e1f6719-f4ec-4c1e-96c3-29de41a22219-model-cache\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 05:00:35.107781 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:35.107740 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e1f6719-f4ec-4c1e-96c3-29de41a22219-home\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 05:00:35.107781 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:35.107755 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e1f6719-f4ec-4c1e-96c3-29de41a22219-tmp-dir\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 05:00:35.109516 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:35.109487 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e1f6719-f4ec-4c1e-96c3-29de41a22219-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e1f6719-f4ec-4c1e-96c3-29de41a22219" (UID: "4e1f6719-f4ec-4c1e-96c3-29de41a22219"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 05:00:35.109635 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:35.109561 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1f6719-f4ec-4c1e-96c3-29de41a22219-dshm" (OuterVolumeSpecName: "dshm") pod "4e1f6719-f4ec-4c1e-96c3-29de41a22219" (UID: "4e1f6719-f4ec-4c1e-96c3-29de41a22219"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 05:00:35.109635 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:35.109565 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e1f6719-f4ec-4c1e-96c3-29de41a22219-kube-api-access-xzxq9" (OuterVolumeSpecName: "kube-api-access-xzxq9") pod "4e1f6719-f4ec-4c1e-96c3-29de41a22219" (UID: "4e1f6719-f4ec-4c1e-96c3-29de41a22219"). InnerVolumeSpecName "kube-api-access-xzxq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 05:00:35.208880 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:35.208841 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e1f6719-f4ec-4c1e-96c3-29de41a22219-dshm\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 05:00:35.208880 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:35.208871 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e1f6719-f4ec-4c1e-96c3-29de41a22219-tls-certs\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 05:00:35.208880 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:35.208886 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzxq9\" (UniqueName: \"kubernetes.io/projected/4e1f6719-f4ec-4c1e-96c3-29de41a22219-kube-api-access-xzxq9\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 05:00:35.958478 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:35.958447 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" Jul 29 05:00:35.958921 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:35.958482 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw" event={"ID":"4e1f6719-f4ec-4c1e-96c3-29de41a22219","Type":"ContainerDied","Data":"1e38633c7f46484c813f5d8e81fb9f80fd2e6f9cd6d5fb5172ee057b170a96a1"} Jul 29 05:00:35.958921 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:35.958532 2579 scope.go:117] "RemoveContainer" containerID="cbc317a29a735eb01aa4bf2127771ee1ef957d673307bbcfa616a5829c0d897e" Jul 29 05:00:35.974457 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:35.974430 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw"] Jul 29 05:00:35.977901 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:35.977875 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-6eef04/sub-test-model-6eef04-kserve-79c5cbcddc-2cczw"] Jul 29 05:00:37.591921 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:37.591884 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e1f6719-f4ec-4c1e-96c3-29de41a22219" path="/var/lib/kubelet/pods/4e1f6719-f4ec-4c1e-96c3-29de41a22219/volumes" Jul 29 05:00:39.963521 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:00:39.963488 2579 scope.go:117] "RemoveContainer" containerID="54595acf85e21fcf621493040ea2782f444b7360a93fc840d4f8d8a3f3b9b4e0" Jul 29 05:02:51.701295 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:51.701258 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c210b8-6586c55556-wfhhb"] Jul 29 05:02:51.701814 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:51.701795 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e1f6719-f4ec-4c1e-96c3-29de41a22219" containerName="main" Jul 29 05:02:51.701880 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:51.701816 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e1f6719-f4ec-4c1e-96c3-29de41a22219" containerName="main" Jul 29 05:02:51.701924 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:51.701883 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e1f6719-f4ec-4c1e-96c3-29de41a22219" containerName="main" Jul 29 05:02:51.705025 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:51.705009 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c210b8-6586c55556-wfhhb" Jul 29 05:02:51.709575 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:51.709550 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c210b8-dockercfg-6mpcc\"" Jul 29 05:02:51.709819 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:51.709805 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c210b8\"" Jul 29 05:02:51.719105 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:51.719082 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c210b8-6586c55556-wfhhb"] Jul 29 05:02:51.734244 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:51.734216 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c210b8-6f6475f8c4-qh8jc"] Jul 29 05:02:51.737570 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:51.737552 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c210b8-6f6475f8c4-qh8jc" Jul 29 05:02:51.747149 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:51.747126 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c210b8-6f6475f8c4-qh8jc"] Jul 29 05:02:51.845715 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:51.845674 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlk95\" (UniqueName: \"kubernetes.io/projected/abef8972-03e1-4b93-bc01-b3649f8ea19b-kube-api-access-jlk95\") pod \"payload-pre-processing-e2e-ipp-a-c210b8-6586c55556-wfhhb\" (UID: \"abef8972-03e1-4b93-bc01-b3649f8ea19b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c210b8-6586c55556-wfhhb" Jul 29 05:02:51.845715 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:51.845718 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c95a4a6-e48a-469e-a437-05e9c350d594-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c210b8-6f6475f8c4-qh8jc\" (UID: \"5c95a4a6-e48a-469e-a437-05e9c350d594\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c210b8-6f6475f8c4-qh8jc" Jul 29 05:02:51.845990 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:51.845758 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abef8972-03e1-4b93-bc01-b3649f8ea19b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c210b8-6586c55556-wfhhb\" (UID: \"abef8972-03e1-4b93-bc01-b3649f8ea19b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c210b8-6586c55556-wfhhb" Jul 29 05:02:51.845990 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:51.845812 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lcss\" (UniqueName: \"kubernetes.io/projected/5c95a4a6-e48a-469e-a437-05e9c350d594-kube-api-access-7lcss\") pod \"payload-processing-e2e-ipp-a-c210b8-6f6475f8c4-qh8jc\" (UID: \"5c95a4a6-e48a-469e-a437-05e9c350d594\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c210b8-6f6475f8c4-qh8jc" Jul 29 05:02:51.946429 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:51.946390 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c95a4a6-e48a-469e-a437-05e9c350d594-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c210b8-6f6475f8c4-qh8jc\" (UID: \"5c95a4a6-e48a-469e-a437-05e9c350d594\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c210b8-6f6475f8c4-qh8jc" Jul 29 05:02:51.946615 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:51.946464 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abef8972-03e1-4b93-bc01-b3649f8ea19b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c210b8-6586c55556-wfhhb\" (UID: \"abef8972-03e1-4b93-bc01-b3649f8ea19b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c210b8-6586c55556-wfhhb" Jul 29 05:02:51.946615 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:51.946522 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lcss\" (UniqueName: \"kubernetes.io/projected/5c95a4a6-e48a-469e-a437-05e9c350d594-kube-api-access-7lcss\") pod \"payload-processing-e2e-ipp-a-c210b8-6f6475f8c4-qh8jc\" (UID: \"5c95a4a6-e48a-469e-a437-05e9c350d594\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c210b8-6f6475f8c4-qh8jc" Jul 29 05:02:51.946615 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:51.946576 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlk95\" (UniqueName: \"kubernetes.io/projected/abef8972-03e1-4b93-bc01-b3649f8ea19b-kube-api-access-jlk95\") pod \"payload-pre-processing-e2e-ipp-a-c210b8-6586c55556-wfhhb\" (UID: \"abef8972-03e1-4b93-bc01-b3649f8ea19b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c210b8-6586c55556-wfhhb" Jul 29 05:02:51.947171 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:51.947149 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c95a4a6-e48a-469e-a437-05e9c350d594-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c210b8-6f6475f8c4-qh8jc\" (UID: \"5c95a4a6-e48a-469e-a437-05e9c350d594\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c210b8-6f6475f8c4-qh8jc" Jul 29 05:02:51.947245 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:51.947198 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abef8972-03e1-4b93-bc01-b3649f8ea19b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c210b8-6586c55556-wfhhb\" (UID: \"abef8972-03e1-4b93-bc01-b3649f8ea19b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c210b8-6586c55556-wfhhb" Jul 29 05:02:51.955016 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:51.954935 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lcss\" (UniqueName: \"kubernetes.io/projected/5c95a4a6-e48a-469e-a437-05e9c350d594-kube-api-access-7lcss\") pod \"payload-processing-e2e-ipp-a-c210b8-6f6475f8c4-qh8jc\" (UID: \"5c95a4a6-e48a-469e-a437-05e9c350d594\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c210b8-6f6475f8c4-qh8jc" Jul 29 05:02:51.955169 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:51.955113 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlk95\" (UniqueName: \"kubernetes.io/projected/abef8972-03e1-4b93-bc01-b3649f8ea19b-kube-api-access-jlk95\") pod \"payload-pre-processing-e2e-ipp-a-c210b8-6586c55556-wfhhb\" (UID: \"abef8972-03e1-4b93-bc01-b3649f8ea19b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c210b8-6586c55556-wfhhb" Jul 29 05:02:52.015313 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:52.015273 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c210b8-6586c55556-wfhhb" Jul 29 05:02:52.047788 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:52.047755 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c210b8-6f6475f8c4-qh8jc" Jul 29 05:02:52.179965 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:52.179917 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c210b8-6586c55556-wfhhb"] Jul 29 05:02:52.182106 ip-10-0-139-111 kubenswrapper[2579]: W0729 05:02:52.182079 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabef8972_03e1_4b93_bc01_b3649f8ea19b.slice/crio-4848f0132d209bc91615ec3f8c53dc173dde4f56f62956b402649aef4e424c79 WatchSource:0}: Error finding container 4848f0132d209bc91615ec3f8c53dc173dde4f56f62956b402649aef4e424c79: Status 404 returned error can't find the container with id 4848f0132d209bc91615ec3f8c53dc173dde4f56f62956b402649aef4e424c79 Jul 29 05:02:52.412624 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:52.412599 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c210b8-6f6475f8c4-qh8jc"] Jul 29 05:02:52.414660 ip-10-0-139-111 kubenswrapper[2579]: W0729 05:02:52.414633 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c95a4a6_e48a_469e_a437_05e9c350d594.slice/crio-8b522c0b12f26b2b943fc8403985bf4b6e68156279270f5338eccd3684824c35 WatchSource:0}: Error finding container 8b522c0b12f26b2b943fc8403985bf4b6e68156279270f5338eccd3684824c35: Status 404 returned error can't find the container with id 8b522c0b12f26b2b943fc8403985bf4b6e68156279270f5338eccd3684824c35 Jul 29 05:02:52.640023 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:52.639892 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c210b8-6586c55556-wfhhb" event={"ID":"abef8972-03e1-4b93-bc01-b3649f8ea19b","Type":"ContainerStarted","Data":"40bdc6e42235adae6af5f61a1359c7d2068722afd60b17890643a527556a849a"} Jul 29 05:02:52.640023 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:52.639959 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c210b8-6586c55556-wfhhb" event={"ID":"abef8972-03e1-4b93-bc01-b3649f8ea19b","Type":"ContainerStarted","Data":"4848f0132d209bc91615ec3f8c53dc173dde4f56f62956b402649aef4e424c79"} Jul 29 05:02:52.640023 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:52.639990 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c210b8-6586c55556-wfhhb" Jul 29 05:02:52.641646 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:52.641618 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c210b8-6f6475f8c4-qh8jc" event={"ID":"5c95a4a6-e48a-469e-a437-05e9c350d594","Type":"ContainerStarted","Data":"d344dcfd80bb10d1f76aa3c9d6f9452e80d1740267022964ee241998dda85cce"} Jul 29 05:02:52.641764 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:52.641668 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c210b8-6f6475f8c4-qh8jc" event={"ID":"5c95a4a6-e48a-469e-a437-05e9c350d594","Type":"ContainerStarted","Data":"8b522c0b12f26b2b943fc8403985bf4b6e68156279270f5338eccd3684824c35"} Jul 29 05:02:52.641764 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:52.641716 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c210b8-6f6475f8c4-qh8jc" Jul 29 05:02:52.661153 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:52.661108 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c210b8-6586c55556-wfhhb" podStartSLOduration=1.66109377 podStartE2EDuration="1.66109377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 05:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 05:02:52.654836024 +0000 UTC m=+3553.514210748" watchObservedRunningTime="2026-07-29 05:02:52.66109377 +0000 UTC m=+3553.520468496" Jul 29 05:02:52.673032 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:02:52.672981 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c210b8-6f6475f8c4-qh8jc" podStartSLOduration=1.672966142 podStartE2EDuration="1.672966142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 05:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 05:02:52.672704808 +0000 UTC m=+3553.532079528" watchObservedRunningTime="2026-07-29 05:02:52.672966142 +0000 UTC m=+3553.532340866" Jul 29 05:03:03.649264 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:03:03.649228 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c210b8-6f6475f8c4-qh8jc" Jul 29 05:03:03.649755 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:03:03.649415 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c210b8-6586c55556-wfhhb" Jul 29 05:03:21.007292 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:03:21.007247 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b1dab2-78997cb8d8-fxc6l"] Jul 29 05:03:21.012173 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:03:21.012149 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1dab2-78997cb8d8-fxc6l" Jul 29 05:03:21.014489 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:03:21.014468 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b1dab2-dockercfg-wbdf2\"" Jul 29 05:03:21.014616 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:03:21.014470 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b1dab2\"" Jul 29 05:03:21.019511 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:03:21.019484 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b1dab2-78997cb8d8-fxc6l"] Jul 29 05:03:21.093032 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:03:21.092993 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szwmb\" (UniqueName: \"kubernetes.io/projected/de6fdfe2-9d65-48fc-80c3-b571655a2a4c-kube-api-access-szwmb\") pod \"payload-pre-processing-e2e-ipp-b-b1dab2-78997cb8d8-fxc6l\" (UID: \"de6fdfe2-9d65-48fc-80c3-b571655a2a4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1dab2-78997cb8d8-fxc6l" Jul 29 05:03:21.093231 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:03:21.093055 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de6fdfe2-9d65-48fc-80c3-b571655a2a4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b1dab2-78997cb8d8-fxc6l\" (UID: \"de6fdfe2-9d65-48fc-80c3-b571655a2a4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1dab2-78997cb8d8-fxc6l" Jul 29 05:03:21.193857 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:03:21.193814 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szwmb\" (UniqueName: \"kubernetes.io/projected/de6fdfe2-9d65-48fc-80c3-b571655a2a4c-kube-api-access-szwmb\") pod \"payload-pre-processing-e2e-ipp-b-b1dab2-78997cb8d8-fxc6l\" (UID: \"de6fdfe2-9d65-48fc-80c3-b571655a2a4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1dab2-78997cb8d8-fxc6l" Jul 29 05:03:21.194066 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:03:21.193898 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de6fdfe2-9d65-48fc-80c3-b571655a2a4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b1dab2-78997cb8d8-fxc6l\" (UID: \"de6fdfe2-9d65-48fc-80c3-b571655a2a4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1dab2-78997cb8d8-fxc6l" Jul 29 05:03:21.194677 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:03:21.194653 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de6fdfe2-9d65-48fc-80c3-b571655a2a4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b1dab2-78997cb8d8-fxc6l\" (UID: \"de6fdfe2-9d65-48fc-80c3-b571655a2a4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1dab2-78997cb8d8-fxc6l" Jul 29 05:03:21.202186 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:03:21.202160 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szwmb\" (UniqueName: \"kubernetes.io/projected/de6fdfe2-9d65-48fc-80c3-b571655a2a4c-kube-api-access-szwmb\") pod \"payload-pre-processing-e2e-ipp-b-b1dab2-78997cb8d8-fxc6l\" (UID: \"de6fdfe2-9d65-48fc-80c3-b571655a2a4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1dab2-78997cb8d8-fxc6l" Jul 29 05:03:21.324960 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:03:21.324864 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1dab2-78997cb8d8-fxc6l" Jul 29 05:03:21.460014 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:03:21.459988 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b1dab2-78997cb8d8-fxc6l"] Jul 29 05:03:21.461844 ip-10-0-139-111 kubenswrapper[2579]: W0729 05:03:21.461809 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde6fdfe2_9d65_48fc_80c3_b571655a2a4c.slice/crio-b6b12b2a6cd099dd1df0cef89f5db55d6e63142817d34af018c8c50f04e324f8 WatchSource:0}: Error finding container b6b12b2a6cd099dd1df0cef89f5db55d6e63142817d34af018c8c50f04e324f8: Status 404 returned error can't find the container with id b6b12b2a6cd099dd1df0cef89f5db55d6e63142817d34af018c8c50f04e324f8 Jul 29 05:03:21.786697 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:03:21.786659 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1dab2-78997cb8d8-fxc6l" event={"ID":"de6fdfe2-9d65-48fc-80c3-b571655a2a4c","Type":"ContainerStarted","Data":"61b54a90f60d98d50cac4264dee42727f503b97463d59f4bf0a27dafc7ca9a5d"} Jul 29 05:03:21.788757 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:03:21.786721 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1dab2-78997cb8d8-fxc6l" event={"ID":"de6fdfe2-9d65-48fc-80c3-b571655a2a4c","Type":"ContainerStarted","Data":"b6b12b2a6cd099dd1df0cef89f5db55d6e63142817d34af018c8c50f04e324f8"} Jul 29 05:03:21.788757 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:03:21.786759 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1dab2-78997cb8d8-fxc6l" Jul 29 05:03:21.802598 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:03:21.802546 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1dab2-78997cb8d8-fxc6l" podStartSLOduration=1.80253079 podStartE2EDuration="1.80253079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 05:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 05:03:21.800504993 +0000 UTC m=+3582.659879723" watchObservedRunningTime="2026-07-29 05:03:21.80253079 +0000 UTC m=+3582.661905514" Jul 29 05:03:32.793951 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:03:32.793869 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1dab2-78997cb8d8-fxc6l" Jul 29 05:03:51.884811 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:03:51.884779 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-vmjvs_8d42dd32-a9fb-49a4-9c1e-34ba3e1a96c0/payload-processing/0.log" Jul 29 05:04:49.166560 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:04:49.166532 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-c210b8-6f6475f8c4-qh8jc_5c95a4a6-e48a-469e-a437-05e9c350d594/payload-processing/0.log" Jul 29 05:04:49.297325 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:04:49.297292 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-vmjvs_8d42dd32-a9fb-49a4-9c1e-34ba3e1a96c0/payload-processing/0.log" Jul 29 05:04:52.910087 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:04:52.910049 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z"] Jul 29 05:04:52.913904 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:04:52.913880 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z" Jul 29 05:04:52.917453 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:04:52.917431 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-aa798b-serving-cert\"" Jul 29 05:04:52.933315 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:04:52.933286 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z"] Jul 29 05:04:53.027334 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:04:53.027296 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b96xj\" (UniqueName: \"kubernetes.io/projected/79f666dc-1e82-42b5-90fc-6e07e17af8b6-kube-api-access-b96xj\") pod \"maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z\" (UID: \"79f666dc-1e82-42b5-90fc-6e07e17af8b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z" Jul 29 05:04:53.027527 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:04:53.027342 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79f666dc-1e82-42b5-90fc-6e07e17af8b6-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z\" (UID: \"79f666dc-1e82-42b5-90fc-6e07e17af8b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z" Jul 29 05:04:53.027527 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:04:53.027431 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79f666dc-1e82-42b5-90fc-6e07e17af8b6-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z\" (UID: \"79f666dc-1e82-42b5-90fc-6e07e17af8b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z" Jul 29 05:04:53.128101 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:04:53.128065 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79f666dc-1e82-42b5-90fc-6e07e17af8b6-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z\" (UID: \"79f666dc-1e82-42b5-90fc-6e07e17af8b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z" Jul 29 05:04:53.128302 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:04:53.128147 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b96xj\" (UniqueName: \"kubernetes.io/projected/79f666dc-1e82-42b5-90fc-6e07e17af8b6-kube-api-access-b96xj\") pod \"maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z\" (UID: \"79f666dc-1e82-42b5-90fc-6e07e17af8b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z" Jul 29 05:04:53.128302 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:04:53.128170 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79f666dc-1e82-42b5-90fc-6e07e17af8b6-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z\" (UID: \"79f666dc-1e82-42b5-90fc-6e07e17af8b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z" Jul 29 05:04:53.128886 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:04:53.128857 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79f666dc-1e82-42b5-90fc-6e07e17af8b6-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z\" (UID: \"79f666dc-1e82-42b5-90fc-6e07e17af8b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z" Jul 29 05:04:53.130591 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:04:53.130568 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79f666dc-1e82-42b5-90fc-6e07e17af8b6-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z\" (UID: \"79f666dc-1e82-42b5-90fc-6e07e17af8b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z" Jul 29 05:04:53.167471 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:04:53.167399 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b96xj\" (UniqueName: \"kubernetes.io/projected/79f666dc-1e82-42b5-90fc-6e07e17af8b6-kube-api-access-b96xj\") pod \"maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z\" (UID: \"79f666dc-1e82-42b5-90fc-6e07e17af8b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z" Jul 29 05:04:53.225299 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:04:53.225270 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z" Jul 29 05:04:53.387871 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:04:53.387837 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z"] Jul 29 05:04:53.389334 ip-10-0-139-111 kubenswrapper[2579]: W0729 05:04:53.389311 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f666dc_1e82_42b5_90fc_6e07e17af8b6.slice/crio-1ba4be18312ac28d9394eaec61d39198431e517968ac3a797728d32bca296701 WatchSource:0}: Error finding container 1ba4be18312ac28d9394eaec61d39198431e517968ac3a797728d32bca296701: Status 404 returned error can't find the container with id 1ba4be18312ac28d9394eaec61d39198431e517968ac3a797728d32bca296701 Jul 29 05:04:53.390720 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:04:53.390702 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 05:04:54.262512 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:04:54.262479 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z" event={"ID":"79f666dc-1e82-42b5-90fc-6e07e17af8b6","Type":"ContainerStarted","Data":"8eec5fe78aa59a8d552e816d1ce31c7711bdb0f6210cf1cdd9988f14f6a1b347"} Jul 29 05:04:54.263013 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:04:54.262538 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z" event={"ID":"79f666dc-1e82-42b5-90fc-6e07e17af8b6","Type":"ContainerStarted","Data":"1ba4be18312ac28d9394eaec61d39198431e517968ac3a797728d32bca296701"} Jul 29 05:04:54.263013 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:04:54.262582 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z" Jul 29 05:04:54.294429 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:04:54.294376 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z" podStartSLOduration=1.827991542 podStartE2EDuration="2.294359121s" totalImagesPullingTime="466.367579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 05:04:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 05:04:54.290396162 +0000 UTC m=+3675.149770908" watchObservedRunningTime="2026-07-29 05:04:54.294359121 +0000 UTC m=+3675.153733845" Jul 29 05:05:00.273372 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:00.273346 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z" Jul 29 05:05:23.952709 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:23.952625 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z"] Jul 29 05:05:23.953178 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:23.952817 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z" podUID="79f666dc-1e82-42b5-90fc-6e07e17af8b6" containerName="maas-api" containerID="cri-o://8eec5fe78aa59a8d552e816d1ce31c7711bdb0f6210cf1cdd9988f14f6a1b347" gracePeriod=30 Jul 29 05:05:24.214793 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:24.214728 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z" Jul 29 05:05:24.333385 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:24.333351 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79f666dc-1e82-42b5-90fc-6e07e17af8b6-openshift-service-ca\") pod \"79f666dc-1e82-42b5-90fc-6e07e17af8b6\" (UID: \"79f666dc-1e82-42b5-90fc-6e07e17af8b6\") " Jul 29 05:05:24.333565 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:24.333421 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b96xj\" (UniqueName: \"kubernetes.io/projected/79f666dc-1e82-42b5-90fc-6e07e17af8b6-kube-api-access-b96xj\") pod \"79f666dc-1e82-42b5-90fc-6e07e17af8b6\" (UID: \"79f666dc-1e82-42b5-90fc-6e07e17af8b6\") " Jul 29 05:05:24.333612 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:24.333570 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79f666dc-1e82-42b5-90fc-6e07e17af8b6-maas-api-tls\") pod \"79f666dc-1e82-42b5-90fc-6e07e17af8b6\" (UID: \"79f666dc-1e82-42b5-90fc-6e07e17af8b6\") " Jul 29 05:05:24.333795 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:24.333769 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79f666dc-1e82-42b5-90fc-6e07e17af8b6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "79f666dc-1e82-42b5-90fc-6e07e17af8b6" (UID: "79f666dc-1e82-42b5-90fc-6e07e17af8b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 05:05:24.335672 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:24.335647 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f666dc-1e82-42b5-90fc-6e07e17af8b6-kube-api-access-b96xj" (OuterVolumeSpecName: "kube-api-access-b96xj") pod "79f666dc-1e82-42b5-90fc-6e07e17af8b6" (UID: "79f666dc-1e82-42b5-90fc-6e07e17af8b6"). InnerVolumeSpecName "kube-api-access-b96xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 05:05:24.335790 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:24.335703 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79f666dc-1e82-42b5-90fc-6e07e17af8b6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "79f666dc-1e82-42b5-90fc-6e07e17af8b6" (UID: "79f666dc-1e82-42b5-90fc-6e07e17af8b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 05:05:24.419076 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:24.419040 2579 generic.go:356] "Generic (PLEG): container finished" podID="79f666dc-1e82-42b5-90fc-6e07e17af8b6" containerID="8eec5fe78aa59a8d552e816d1ce31c7711bdb0f6210cf1cdd9988f14f6a1b347" exitCode=0 Jul 29 05:05:24.419276 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:24.419084 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z" Jul 29 05:05:24.419276 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:24.419128 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z" event={"ID":"79f666dc-1e82-42b5-90fc-6e07e17af8b6","Type":"ContainerDied","Data":"8eec5fe78aa59a8d552e816d1ce31c7711bdb0f6210cf1cdd9988f14f6a1b347"} Jul 29 05:05:24.419276 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:24.419178 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z" event={"ID":"79f666dc-1e82-42b5-90fc-6e07e17af8b6","Type":"ContainerDied","Data":"1ba4be18312ac28d9394eaec61d39198431e517968ac3a797728d32bca296701"} Jul 29 05:05:24.419276 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:24.419227 2579 scope.go:117] "RemoveContainer" containerID="8eec5fe78aa59a8d552e816d1ce31c7711bdb0f6210cf1cdd9988f14f6a1b347" Jul 29 05:05:24.429233 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:24.429216 2579 scope.go:117] "RemoveContainer" containerID="8eec5fe78aa59a8d552e816d1ce31c7711bdb0f6210cf1cdd9988f14f6a1b347" Jul 29 05:05:24.429550 ip-10-0-139-111 kubenswrapper[2579]: E0729 05:05:24.429531 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eec5fe78aa59a8d552e816d1ce31c7711bdb0f6210cf1cdd9988f14f6a1b347\": container with ID starting with 8eec5fe78aa59a8d552e816d1ce31c7711bdb0f6210cf1cdd9988f14f6a1b347 not found: ID does not exist" containerID="8eec5fe78aa59a8d552e816d1ce31c7711bdb0f6210cf1cdd9988f14f6a1b347" Jul 29 05:05:24.429623 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:24.429564 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eec5fe78aa59a8d552e816d1ce31c7711bdb0f6210cf1cdd9988f14f6a1b347"} err="failed to get container status \"8eec5fe78aa59a8d552e816d1ce31c7711bdb0f6210cf1cdd9988f14f6a1b347\": rpc error: code = NotFound desc = could not find container \"8eec5fe78aa59a8d552e816d1ce31c7711bdb0f6210cf1cdd9988f14f6a1b347\": container with ID starting with 8eec5fe78aa59a8d552e816d1ce31c7711bdb0f6210cf1cdd9988f14f6a1b347 not found: ID does not exist" Jul 29 05:05:24.434269 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:24.434247 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79f666dc-1e82-42b5-90fc-6e07e17af8b6-maas-api-tls\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 05:05:24.434269 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:24.434268 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79f666dc-1e82-42b5-90fc-6e07e17af8b6-openshift-service-ca\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 05:05:24.434429 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:24.434278 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b96xj\" (UniqueName: \"kubernetes.io/projected/79f666dc-1e82-42b5-90fc-6e07e17af8b6-kube-api-access-b96xj\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 05:05:24.449968 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:24.449935 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z"] Jul 29 05:05:24.460491 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:24.460462 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-aa798b-68fd85d7b7-6j65z"] Jul 29 05:05:25.591624 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:25.591582 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79f666dc-1e82-42b5-90fc-6e07e17af8b6" path="/var/lib/kubelet/pods/79f666dc-1e82-42b5-90fc-6e07e17af8b6/volumes" Jul 29 05:05:51.143998 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.143958 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c210b8-6f6475f8c4-qh8jc"] Jul 29 05:05:51.144551 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.144233 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c210b8-6f6475f8c4-qh8jc" podUID="5c95a4a6-e48a-469e-a437-05e9c350d594" containerName="payload-processing" containerID="cri-o://d344dcfd80bb10d1f76aa3c9d6f9452e80d1740267022964ee241998dda85cce" gracePeriod=30 Jul 29 05:05:51.153586 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.153557 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c210b8-6586c55556-wfhhb"] Jul 29 05:05:51.153801 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.153775 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c210b8-6586c55556-wfhhb" podUID="abef8972-03e1-4b93-bc01-b3649f8ea19b" containerName="payload-pre-processing" containerID="cri-o://40bdc6e42235adae6af5f61a1359c7d2068722afd60b17890643a527556a849a" gracePeriod=30 Jul 29 05:05:51.447373 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.447344 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c210b8-6f6475f8c4-qh8jc" Jul 29 05:05:51.450720 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.450695 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c210b8-6586c55556-wfhhb" Jul 29 05:05:51.470599 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.470567 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c95a4a6-e48a-469e-a437-05e9c350d594-plugins-config-volume\") pod \"5c95a4a6-e48a-469e-a437-05e9c350d594\" (UID: \"5c95a4a6-e48a-469e-a437-05e9c350d594\") " Jul 29 05:05:51.470782 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.470710 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lcss\" (UniqueName: \"kubernetes.io/projected/5c95a4a6-e48a-469e-a437-05e9c350d594-kube-api-access-7lcss\") pod \"5c95a4a6-e48a-469e-a437-05e9c350d594\" (UID: \"5c95a4a6-e48a-469e-a437-05e9c350d594\") " Jul 29 05:05:51.471245 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.471075 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c95a4a6-e48a-469e-a437-05e9c350d594-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5c95a4a6-e48a-469e-a437-05e9c350d594" (UID: "5c95a4a6-e48a-469e-a437-05e9c350d594"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 05:05:51.473509 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.473449 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c95a4a6-e48a-469e-a437-05e9c350d594-kube-api-access-7lcss" (OuterVolumeSpecName: "kube-api-access-7lcss") pod "5c95a4a6-e48a-469e-a437-05e9c350d594" (UID: "5c95a4a6-e48a-469e-a437-05e9c350d594"). InnerVolumeSpecName "kube-api-access-7lcss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 05:05:51.565566 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.565534 2579 generic.go:356] "Generic (PLEG): container finished" podID="abef8972-03e1-4b93-bc01-b3649f8ea19b" containerID="40bdc6e42235adae6af5f61a1359c7d2068722afd60b17890643a527556a849a" exitCode=0 Jul 29 05:05:51.565753 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.565583 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c210b8-6586c55556-wfhhb" Jul 29 05:05:51.565753 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.565622 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c210b8-6586c55556-wfhhb" event={"ID":"abef8972-03e1-4b93-bc01-b3649f8ea19b","Type":"ContainerDied","Data":"40bdc6e42235adae6af5f61a1359c7d2068722afd60b17890643a527556a849a"} Jul 29 05:05:51.565753 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.565675 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c210b8-6586c55556-wfhhb" event={"ID":"abef8972-03e1-4b93-bc01-b3649f8ea19b","Type":"ContainerDied","Data":"4848f0132d209bc91615ec3f8c53dc173dde4f56f62956b402649aef4e424c79"} Jul 29 05:05:51.565753 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.565708 2579 scope.go:117] "RemoveContainer" containerID="40bdc6e42235adae6af5f61a1359c7d2068722afd60b17890643a527556a849a" Jul 29 05:05:51.566971 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.566946 2579 generic.go:356] "Generic (PLEG): container finished" podID="5c95a4a6-e48a-469e-a437-05e9c350d594" containerID="d344dcfd80bb10d1f76aa3c9d6f9452e80d1740267022964ee241998dda85cce" exitCode=0 Jul 29 05:05:51.567091 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.566982 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c210b8-6f6475f8c4-qh8jc" event={"ID":"5c95a4a6-e48a-469e-a437-05e9c350d594","Type":"ContainerDied","Data":"d344dcfd80bb10d1f76aa3c9d6f9452e80d1740267022964ee241998dda85cce"} Jul 29 05:05:51.567091 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.567020 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c210b8-6f6475f8c4-qh8jc" event={"ID":"5c95a4a6-e48a-469e-a437-05e9c350d594","Type":"ContainerDied","Data":"8b522c0b12f26b2b943fc8403985bf4b6e68156279270f5338eccd3684824c35"} Jul 29 05:05:51.567091 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.566988 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c210b8-6f6475f8c4-qh8jc" Jul 29 05:05:51.571226 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.571176 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abef8972-03e1-4b93-bc01-b3649f8ea19b-plugins-config-volume\") pod \"abef8972-03e1-4b93-bc01-b3649f8ea19b\" (UID: \"abef8972-03e1-4b93-bc01-b3649f8ea19b\") " Jul 29 05:05:51.571370 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.571264 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlk95\" (UniqueName: \"kubernetes.io/projected/abef8972-03e1-4b93-bc01-b3649f8ea19b-kube-api-access-jlk95\") pod \"abef8972-03e1-4b93-bc01-b3649f8ea19b\" (UID: \"abef8972-03e1-4b93-bc01-b3649f8ea19b\") " Jul 29 05:05:51.571528 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.571494 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abef8972-03e1-4b93-bc01-b3649f8ea19b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "abef8972-03e1-4b93-bc01-b3649f8ea19b" (UID: "abef8972-03e1-4b93-bc01-b3649f8ea19b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 05:05:51.571588 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.571575 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lcss\" (UniqueName: \"kubernetes.io/projected/5c95a4a6-e48a-469e-a437-05e9c350d594-kube-api-access-7lcss\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 05:05:51.571639 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.571590 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abef8972-03e1-4b93-bc01-b3649f8ea19b-plugins-config-volume\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 05:05:51.571639 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.571603 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c95a4a6-e48a-469e-a437-05e9c350d594-plugins-config-volume\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 05:05:51.573101 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.573081 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abef8972-03e1-4b93-bc01-b3649f8ea19b-kube-api-access-jlk95" (OuterVolumeSpecName: "kube-api-access-jlk95") pod "abef8972-03e1-4b93-bc01-b3649f8ea19b" (UID: "abef8972-03e1-4b93-bc01-b3649f8ea19b"). InnerVolumeSpecName "kube-api-access-jlk95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 05:05:51.575924 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.575899 2579 scope.go:117] "RemoveContainer" containerID="40bdc6e42235adae6af5f61a1359c7d2068722afd60b17890643a527556a849a" Jul 29 05:05:51.576238 ip-10-0-139-111 kubenswrapper[2579]: E0729 05:05:51.576191 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40bdc6e42235adae6af5f61a1359c7d2068722afd60b17890643a527556a849a\": container with ID starting with 40bdc6e42235adae6af5f61a1359c7d2068722afd60b17890643a527556a849a not found: ID does not exist" containerID="40bdc6e42235adae6af5f61a1359c7d2068722afd60b17890643a527556a849a" Jul 29 05:05:51.576321 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.576247 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40bdc6e42235adae6af5f61a1359c7d2068722afd60b17890643a527556a849a"} err="failed to get container status \"40bdc6e42235adae6af5f61a1359c7d2068722afd60b17890643a527556a849a\": rpc error: code = NotFound desc = could not find container \"40bdc6e42235adae6af5f61a1359c7d2068722afd60b17890643a527556a849a\": container with ID starting with 40bdc6e42235adae6af5f61a1359c7d2068722afd60b17890643a527556a849a not found: ID does not exist" Jul 29 05:05:51.576321 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.576265 2579 scope.go:117] "RemoveContainer" containerID="d344dcfd80bb10d1f76aa3c9d6f9452e80d1740267022964ee241998dda85cce" Jul 29 05:05:51.584503 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.584479 2579 scope.go:117] "RemoveContainer" containerID="d344dcfd80bb10d1f76aa3c9d6f9452e80d1740267022964ee241998dda85cce" Jul 29 05:05:51.584764 ip-10-0-139-111 kubenswrapper[2579]: E0729 05:05:51.584745 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d344dcfd80bb10d1f76aa3c9d6f9452e80d1740267022964ee241998dda85cce\": container with ID starting with d344dcfd80bb10d1f76aa3c9d6f9452e80d1740267022964ee241998dda85cce not found: ID does not exist" containerID="d344dcfd80bb10d1f76aa3c9d6f9452e80d1740267022964ee241998dda85cce" Jul 29 05:05:51.584819 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.584773 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d344dcfd80bb10d1f76aa3c9d6f9452e80d1740267022964ee241998dda85cce"} err="failed to get container status \"d344dcfd80bb10d1f76aa3c9d6f9452e80d1740267022964ee241998dda85cce\": rpc error: code = NotFound desc = could not find container \"d344dcfd80bb10d1f76aa3c9d6f9452e80d1740267022964ee241998dda85cce\": container with ID starting with d344dcfd80bb10d1f76aa3c9d6f9452e80d1740267022964ee241998dda85cce not found: ID does not exist" Jul 29 05:05:51.597607 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.597579 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c210b8-6f6475f8c4-qh8jc"] Jul 29 05:05:51.603914 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.603889 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c210b8-6f6475f8c4-qh8jc"] Jul 29 05:05:51.672518 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.672423 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlk95\" (UniqueName: \"kubernetes.io/projected/abef8972-03e1-4b93-bc01-b3649f8ea19b-kube-api-access-jlk95\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 05:05:51.890473 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.890431 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c210b8-6586c55556-wfhhb"] Jul 29 05:05:51.894394 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:51.894369 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c210b8-6586c55556-wfhhb"] Jul 29 05:05:53.596219 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:53.596175 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c95a4a6-e48a-469e-a437-05e9c350d594" path="/var/lib/kubelet/pods/5c95a4a6-e48a-469e-a437-05e9c350d594/volumes" Jul 29 05:05:53.596607 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:05:53.596567 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abef8972-03e1-4b93-bc01-b3649f8ea19b" path="/var/lib/kubelet/pods/abef8972-03e1-4b93-bc01-b3649f8ea19b/volumes" Jul 29 05:06:56.573704 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:06:56.573669 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b1dab2-78997cb8d8-fxc6l"] Jul 29 05:06:56.574326 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:06:56.573931 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1dab2-78997cb8d8-fxc6l" podUID="de6fdfe2-9d65-48fc-80c3-b571655a2a4c" containerName="payload-pre-processing" containerID="cri-o://61b54a90f60d98d50cac4264dee42727f503b97463d59f4bf0a27dafc7ca9a5d" gracePeriod=30 Jul 29 05:06:56.843444 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:06:56.843418 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1dab2-78997cb8d8-fxc6l" Jul 29 05:06:56.902171 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:06:56.902137 2579 generic.go:356] "Generic (PLEG): container finished" podID="de6fdfe2-9d65-48fc-80c3-b571655a2a4c" containerID="61b54a90f60d98d50cac4264dee42727f503b97463d59f4bf0a27dafc7ca9a5d" exitCode=0 Jul 29 05:06:56.902374 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:06:56.902193 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1dab2-78997cb8d8-fxc6l" Jul 29 05:06:56.902374 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:06:56.902252 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1dab2-78997cb8d8-fxc6l" event={"ID":"de6fdfe2-9d65-48fc-80c3-b571655a2a4c","Type":"ContainerDied","Data":"61b54a90f60d98d50cac4264dee42727f503b97463d59f4bf0a27dafc7ca9a5d"} Jul 29 05:06:56.902374 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:06:56.902344 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1dab2-78997cb8d8-fxc6l" event={"ID":"de6fdfe2-9d65-48fc-80c3-b571655a2a4c","Type":"ContainerDied","Data":"b6b12b2a6cd099dd1df0cef89f5db55d6e63142817d34af018c8c50f04e324f8"} Jul 29 05:06:56.902513 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:06:56.902374 2579 scope.go:117] "RemoveContainer" containerID="61b54a90f60d98d50cac4264dee42727f503b97463d59f4bf0a27dafc7ca9a5d" Jul 29 05:06:56.911252 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:06:56.911236 2579 scope.go:117] "RemoveContainer" containerID="61b54a90f60d98d50cac4264dee42727f503b97463d59f4bf0a27dafc7ca9a5d" Jul 29 05:06:56.911505 ip-10-0-139-111 kubenswrapper[2579]: E0729 05:06:56.911487 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61b54a90f60d98d50cac4264dee42727f503b97463d59f4bf0a27dafc7ca9a5d\": container with ID starting with 61b54a90f60d98d50cac4264dee42727f503b97463d59f4bf0a27dafc7ca9a5d not found: ID does not exist" containerID="61b54a90f60d98d50cac4264dee42727f503b97463d59f4bf0a27dafc7ca9a5d" Jul 29 05:06:56.911566 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:06:56.911511 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b54a90f60d98d50cac4264dee42727f503b97463d59f4bf0a27dafc7ca9a5d"} err="failed to get container status \"61b54a90f60d98d50cac4264dee42727f503b97463d59f4bf0a27dafc7ca9a5d\": rpc error: code = NotFound desc = could not find container \"61b54a90f60d98d50cac4264dee42727f503b97463d59f4bf0a27dafc7ca9a5d\": container with ID starting with 61b54a90f60d98d50cac4264dee42727f503b97463d59f4bf0a27dafc7ca9a5d not found: ID does not exist" Jul 29 05:06:56.954072 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:06:56.954038 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szwmb\" (UniqueName: \"kubernetes.io/projected/de6fdfe2-9d65-48fc-80c3-b571655a2a4c-kube-api-access-szwmb\") pod \"de6fdfe2-9d65-48fc-80c3-b571655a2a4c\" (UID: \"de6fdfe2-9d65-48fc-80c3-b571655a2a4c\") " Jul 29 05:06:56.954253 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:06:56.954171 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de6fdfe2-9d65-48fc-80c3-b571655a2a4c-plugins-config-volume\") pod \"de6fdfe2-9d65-48fc-80c3-b571655a2a4c\" (UID: \"de6fdfe2-9d65-48fc-80c3-b571655a2a4c\") " Jul 29 05:06:56.954527 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:06:56.954496 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de6fdfe2-9d65-48fc-80c3-b571655a2a4c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "de6fdfe2-9d65-48fc-80c3-b571655a2a4c" (UID: "de6fdfe2-9d65-48fc-80c3-b571655a2a4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 05:06:56.956136 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:06:56.956116 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de6fdfe2-9d65-48fc-80c3-b571655a2a4c-kube-api-access-szwmb" (OuterVolumeSpecName: "kube-api-access-szwmb") pod "de6fdfe2-9d65-48fc-80c3-b571655a2a4c" (UID: "de6fdfe2-9d65-48fc-80c3-b571655a2a4c"). InnerVolumeSpecName "kube-api-access-szwmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 05:06:57.055054 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:06:57.055018 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szwmb\" (UniqueName: \"kubernetes.io/projected/de6fdfe2-9d65-48fc-80c3-b571655a2a4c-kube-api-access-szwmb\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 05:06:57.055054 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:06:57.055053 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de6fdfe2-9d65-48fc-80c3-b571655a2a4c-plugins-config-volume\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 05:06:57.227762 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:06:57.227735 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b1dab2-78997cb8d8-fxc6l"] Jul 29 05:06:57.231248 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:06:57.231225 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b1dab2-78997cb8d8-fxc6l"] Jul 29 05:06:57.592125 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:06:57.592047 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de6fdfe2-9d65-48fc-80c3-b571655a2a4c" path="/var/lib/kubelet/pods/de6fdfe2-9d65-48fc-80c3-b571655a2a4c/volumes" Jul 29 05:07:08.749060 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:08.749021 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-6eef04-76c4cb4c9d-4l5kj"] Jul 29 05:07:08.749685 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:08.749277 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-6eef04-76c4cb4c9d-4l5kj" podUID="6b6103e8-6946-431b-b34d-5d6b63739df7" containerName="payload-processing" containerID="cri-o://90b46f0303ac3c477802fc73659eb18c0f4c8606e908251c27a47f13ab4672b9" gracePeriod=30 Jul 29 05:07:08.756665 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:08.756634 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-6eef04-8469cf6cb8-5vxlk"] Jul 29 05:07:08.757516 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:08.757130 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6eef04-8469cf6cb8-5vxlk" podUID="453ac86d-ebcd-47d0-97a4-5a94969abffa" containerName="payload-pre-processing" containerID="cri-o://7a4a39456779a105841a58ba71fe28e05a7da43056c398b88d6f2f276caa81f4" gracePeriod=30 Jul 29 05:07:08.971052 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:08.971017 2579 generic.go:356] "Generic (PLEG): container finished" podID="6b6103e8-6946-431b-b34d-5d6b63739df7" containerID="90b46f0303ac3c477802fc73659eb18c0f4c8606e908251c27a47f13ab4672b9" exitCode=0 Jul 29 05:07:08.971339 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:08.971106 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-6eef04-76c4cb4c9d-4l5kj" event={"ID":"6b6103e8-6946-431b-b34d-5d6b63739df7","Type":"ContainerDied","Data":"90b46f0303ac3c477802fc73659eb18c0f4c8606e908251c27a47f13ab4672b9"} Jul 29 05:07:08.972996 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:08.972966 2579 generic.go:356] "Generic (PLEG): container finished" podID="453ac86d-ebcd-47d0-97a4-5a94969abffa" containerID="7a4a39456779a105841a58ba71fe28e05a7da43056c398b88d6f2f276caa81f4" exitCode=0 Jul 29 05:07:08.973111 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:08.973039 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6eef04-8469cf6cb8-5vxlk" event={"ID":"453ac86d-ebcd-47d0-97a4-5a94969abffa","Type":"ContainerDied","Data":"7a4a39456779a105841a58ba71fe28e05a7da43056c398b88d6f2f276caa81f4"} Jul 29 05:07:09.042882 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:09.042856 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-6eef04-76c4cb4c9d-4l5kj" Jul 29 05:07:09.059077 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:09.058996 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6eef04-8469cf6cb8-5vxlk" Jul 29 05:07:09.166606 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:09.166565 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/453ac86d-ebcd-47d0-97a4-5a94969abffa-plugins-config-volume\") pod \"453ac86d-ebcd-47d0-97a4-5a94969abffa\" (UID: \"453ac86d-ebcd-47d0-97a4-5a94969abffa\") " Jul 29 05:07:09.166789 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:09.166667 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b6103e8-6946-431b-b34d-5d6b63739df7-plugins-config-volume\") pod \"6b6103e8-6946-431b-b34d-5d6b63739df7\" (UID: \"6b6103e8-6946-431b-b34d-5d6b63739df7\") " Jul 29 05:07:09.166789 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:09.166733 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqtwz\" (UniqueName: \"kubernetes.io/projected/453ac86d-ebcd-47d0-97a4-5a94969abffa-kube-api-access-gqtwz\") pod \"453ac86d-ebcd-47d0-97a4-5a94969abffa\" (UID: \"453ac86d-ebcd-47d0-97a4-5a94969abffa\") " Jul 29 05:07:09.166789 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:09.166754 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pjtx\" (UniqueName: \"kubernetes.io/projected/6b6103e8-6946-431b-b34d-5d6b63739df7-kube-api-access-9pjtx\") pod \"6b6103e8-6946-431b-b34d-5d6b63739df7\" (UID: \"6b6103e8-6946-431b-b34d-5d6b63739df7\") " Jul 29 05:07:09.167020 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:09.166988 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453ac86d-ebcd-47d0-97a4-5a94969abffa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "453ac86d-ebcd-47d0-97a4-5a94969abffa" (UID: "453ac86d-ebcd-47d0-97a4-5a94969abffa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 05:07:09.167074 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:09.167024 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b6103e8-6946-431b-b34d-5d6b63739df7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6b6103e8-6946-431b-b34d-5d6b63739df7" (UID: "6b6103e8-6946-431b-b34d-5d6b63739df7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 05:07:09.168766 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:09.168738 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6103e8-6946-431b-b34d-5d6b63739df7-kube-api-access-9pjtx" (OuterVolumeSpecName: "kube-api-access-9pjtx") pod "6b6103e8-6946-431b-b34d-5d6b63739df7" (UID: "6b6103e8-6946-431b-b34d-5d6b63739df7"). InnerVolumeSpecName "kube-api-access-9pjtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 05:07:09.168886 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:09.168867 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453ac86d-ebcd-47d0-97a4-5a94969abffa-kube-api-access-gqtwz" (OuterVolumeSpecName: "kube-api-access-gqtwz") pod "453ac86d-ebcd-47d0-97a4-5a94969abffa" (UID: "453ac86d-ebcd-47d0-97a4-5a94969abffa"). InnerVolumeSpecName "kube-api-access-gqtwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 05:07:09.267567 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:09.267533 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b6103e8-6946-431b-b34d-5d6b63739df7-plugins-config-volume\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 05:07:09.267567 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:09.267560 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqtwz\" (UniqueName: \"kubernetes.io/projected/453ac86d-ebcd-47d0-97a4-5a94969abffa-kube-api-access-gqtwz\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 05:07:09.267567 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:09.267570 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pjtx\" (UniqueName: \"kubernetes.io/projected/6b6103e8-6946-431b-b34d-5d6b63739df7-kube-api-access-9pjtx\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 05:07:09.267801 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:09.267579 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/453ac86d-ebcd-47d0-97a4-5a94969abffa-plugins-config-volume\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 05:07:09.979430 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:09.979399 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-6eef04-76c4cb4c9d-4l5kj" Jul 29 05:07:09.979931 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:09.979401 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-6eef04-76c4cb4c9d-4l5kj" event={"ID":"6b6103e8-6946-431b-b34d-5d6b63739df7","Type":"ContainerDied","Data":"edacb7a1db1809c7e2e96e845b21ff97f26a6ebfe6d9e1fc607d2830cccbcff0"} Jul 29 05:07:09.979931 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:09.979556 2579 scope.go:117] "RemoveContainer" containerID="90b46f0303ac3c477802fc73659eb18c0f4c8606e908251c27a47f13ab4672b9" Jul 29 05:07:09.981265 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:09.981244 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6eef04-8469cf6cb8-5vxlk" Jul 29 05:07:09.981265 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:09.981257 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6eef04-8469cf6cb8-5vxlk" event={"ID":"453ac86d-ebcd-47d0-97a4-5a94969abffa","Type":"ContainerDied","Data":"4911cc5eb57d3d2a78a1af19229c363727a361904838a69e2355d3753de81af6"} Jul 29 05:07:09.988858 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:09.988840 2579 scope.go:117] "RemoveContainer" containerID="7a4a39456779a105841a58ba71fe28e05a7da43056c398b88d6f2f276caa81f4" Jul 29 05:07:10.008717 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:10.008688 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-6eef04-8469cf6cb8-5vxlk"] Jul 29 05:07:10.022749 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:10.022722 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-6eef04-8469cf6cb8-5vxlk"] Jul 29 05:07:10.061954 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:10.061923 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-6eef04-76c4cb4c9d-4l5kj"] Jul 29 05:07:10.074201 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:10.074171 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-6eef04-76c4cb4c9d-4l5kj"] Jul 29 05:07:11.591638 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:11.591593 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="453ac86d-ebcd-47d0-97a4-5a94969abffa" path="/var/lib/kubelet/pods/453ac86d-ebcd-47d0-97a4-5a94969abffa/volumes" Jul 29 05:07:11.592169 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:11.592101 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b6103e8-6946-431b-b34d-5d6b63739df7" path="/var/lib/kubelet/pods/6b6103e8-6946-431b-b34d-5d6b63739df7/volumes" Jul 29 05:07:26.643571 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:26.643535 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-be8bc5-6f6c4fd6f8-l882t"] Jul 29 05:07:26.644161 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:26.643802 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-be8bc5-6f6c4fd6f8-l882t" podUID="353191af-c906-4fa7-b567-23bcb9d54318" containerName="payload-processing" containerID="cri-o://6f338524a5f34caaa6af410dc9c98ec8d5e186f2530e6bff1bbc431b65294c7f" gracePeriod=30 Jul 29 05:07:26.661645 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:26.661619 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-be8bc5-57d59449d8-w8h52"] Jul 29 05:07:26.661856 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:26.661832 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-be8bc5-57d59449d8-w8h52" podUID="90926e7a-4e34-473a-af59-d28100545dc0" containerName="payload-pre-processing" containerID="cri-o://c84748e8e6b3d0e79ca0d65205581ef25ffdf675ed4d978818347aa16cd8fd55" gracePeriod=30 Jul 29 05:07:26.930526 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:26.930501 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-be8bc5-6f6c4fd6f8-l882t" Jul 29 05:07:26.950750 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:26.950729 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-be8bc5-57d59449d8-w8h52" Jul 29 05:07:27.002249 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.002198 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9sqmr\" (UniqueName: \"kubernetes.io/projected/353191af-c906-4fa7-b567-23bcb9d54318-kube-api-access-9sqmr\") pod \"353191af-c906-4fa7-b567-23bcb9d54318\" (UID: \"353191af-c906-4fa7-b567-23bcb9d54318\") " Jul 29 05:07:27.002414 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.002265 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqfhz\" (UniqueName: \"kubernetes.io/projected/90926e7a-4e34-473a-af59-d28100545dc0-kube-api-access-sqfhz\") pod \"90926e7a-4e34-473a-af59-d28100545dc0\" (UID: \"90926e7a-4e34-473a-af59-d28100545dc0\") " Jul 29 05:07:27.002414 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.002313 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90926e7a-4e34-473a-af59-d28100545dc0-plugins-config-volume\") pod \"90926e7a-4e34-473a-af59-d28100545dc0\" (UID: \"90926e7a-4e34-473a-af59-d28100545dc0\") " Jul 29 05:07:27.002414 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.002380 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353191af-c906-4fa7-b567-23bcb9d54318-plugins-config-volume\") pod \"353191af-c906-4fa7-b567-23bcb9d54318\" (UID: \"353191af-c906-4fa7-b567-23bcb9d54318\") " Jul 29 05:07:27.002735 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.002711 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90926e7a-4e34-473a-af59-d28100545dc0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "90926e7a-4e34-473a-af59-d28100545dc0" (UID: "90926e7a-4e34-473a-af59-d28100545dc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 05:07:27.002814 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.002713 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/353191af-c906-4fa7-b567-23bcb9d54318-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "353191af-c906-4fa7-b567-23bcb9d54318" (UID: "353191af-c906-4fa7-b567-23bcb9d54318"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 05:07:27.004447 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.004414 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90926e7a-4e34-473a-af59-d28100545dc0-kube-api-access-sqfhz" (OuterVolumeSpecName: "kube-api-access-sqfhz") pod "90926e7a-4e34-473a-af59-d28100545dc0" (UID: "90926e7a-4e34-473a-af59-d28100545dc0"). InnerVolumeSpecName "kube-api-access-sqfhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 05:07:27.004550 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.004488 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/353191af-c906-4fa7-b567-23bcb9d54318-kube-api-access-9sqmr" (OuterVolumeSpecName: "kube-api-access-9sqmr") pod "353191af-c906-4fa7-b567-23bcb9d54318" (UID: "353191af-c906-4fa7-b567-23bcb9d54318"). InnerVolumeSpecName "kube-api-access-9sqmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 05:07:27.069002 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.068965 2579 generic.go:356] "Generic (PLEG): container finished" podID="90926e7a-4e34-473a-af59-d28100545dc0" containerID="c84748e8e6b3d0e79ca0d65205581ef25ffdf675ed4d978818347aa16cd8fd55" exitCode=0 Jul 29 05:07:27.069180 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.069013 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-be8bc5-57d59449d8-w8h52" Jul 29 05:07:27.069180 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.069049 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-be8bc5-57d59449d8-w8h52" event={"ID":"90926e7a-4e34-473a-af59-d28100545dc0","Type":"ContainerDied","Data":"c84748e8e6b3d0e79ca0d65205581ef25ffdf675ed4d978818347aa16cd8fd55"} Jul 29 05:07:27.069180 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.069105 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-be8bc5-57d59449d8-w8h52" event={"ID":"90926e7a-4e34-473a-af59-d28100545dc0","Type":"ContainerDied","Data":"4280c266d0ecb3fd39104c97cbd73e02a4b8f9ea38f50a507c6e910396731a75"} Jul 29 05:07:27.069180 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.069130 2579 scope.go:117] "RemoveContainer" containerID="c84748e8e6b3d0e79ca0d65205581ef25ffdf675ed4d978818347aa16cd8fd55" Jul 29 05:07:27.070664 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.070639 2579 generic.go:356] "Generic (PLEG): container finished" podID="353191af-c906-4fa7-b567-23bcb9d54318" containerID="6f338524a5f34caaa6af410dc9c98ec8d5e186f2530e6bff1bbc431b65294c7f" exitCode=0 Jul 29 05:07:27.070784 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.070686 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-be8bc5-6f6c4fd6f8-l882t" Jul 29 05:07:27.070784 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.070711 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-be8bc5-6f6c4fd6f8-l882t" event={"ID":"353191af-c906-4fa7-b567-23bcb9d54318","Type":"ContainerDied","Data":"6f338524a5f34caaa6af410dc9c98ec8d5e186f2530e6bff1bbc431b65294c7f"} Jul 29 05:07:27.070784 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.070750 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-be8bc5-6f6c4fd6f8-l882t" event={"ID":"353191af-c906-4fa7-b567-23bcb9d54318","Type":"ContainerDied","Data":"fa86d468c772ef618132823785ca25240ee6facd260b464f0595124bfd592325"} Jul 29 05:07:27.078332 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.078170 2579 scope.go:117] "RemoveContainer" containerID="c84748e8e6b3d0e79ca0d65205581ef25ffdf675ed4d978818347aa16cd8fd55" Jul 29 05:07:27.078462 ip-10-0-139-111 kubenswrapper[2579]: E0729 05:07:27.078440 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c84748e8e6b3d0e79ca0d65205581ef25ffdf675ed4d978818347aa16cd8fd55\": container with ID starting with c84748e8e6b3d0e79ca0d65205581ef25ffdf675ed4d978818347aa16cd8fd55 not found: ID does not exist" containerID="c84748e8e6b3d0e79ca0d65205581ef25ffdf675ed4d978818347aa16cd8fd55" Jul 29 05:07:27.078515 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.078469 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c84748e8e6b3d0e79ca0d65205581ef25ffdf675ed4d978818347aa16cd8fd55"} err="failed to get container status \"c84748e8e6b3d0e79ca0d65205581ef25ffdf675ed4d978818347aa16cd8fd55\": rpc error: code = NotFound desc = could not find container \"c84748e8e6b3d0e79ca0d65205581ef25ffdf675ed4d978818347aa16cd8fd55\": container with ID starting with c84748e8e6b3d0e79ca0d65205581ef25ffdf675ed4d978818347aa16cd8fd55 not found: ID does not exist" Jul 29 05:07:27.078515 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.078487 2579 scope.go:117] "RemoveContainer" containerID="6f338524a5f34caaa6af410dc9c98ec8d5e186f2530e6bff1bbc431b65294c7f" Jul 29 05:07:27.086314 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.086294 2579 scope.go:117] "RemoveContainer" containerID="6f338524a5f34caaa6af410dc9c98ec8d5e186f2530e6bff1bbc431b65294c7f" Jul 29 05:07:27.086565 ip-10-0-139-111 kubenswrapper[2579]: E0729 05:07:27.086542 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f338524a5f34caaa6af410dc9c98ec8d5e186f2530e6bff1bbc431b65294c7f\": container with ID starting with 6f338524a5f34caaa6af410dc9c98ec8d5e186f2530e6bff1bbc431b65294c7f not found: ID does not exist" containerID="6f338524a5f34caaa6af410dc9c98ec8d5e186f2530e6bff1bbc431b65294c7f" Jul 29 05:07:27.086656 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.086570 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f338524a5f34caaa6af410dc9c98ec8d5e186f2530e6bff1bbc431b65294c7f"} err="failed to get container status \"6f338524a5f34caaa6af410dc9c98ec8d5e186f2530e6bff1bbc431b65294c7f\": rpc error: code = NotFound desc = could not find container \"6f338524a5f34caaa6af410dc9c98ec8d5e186f2530e6bff1bbc431b65294c7f\": container with ID starting with 6f338524a5f34caaa6af410dc9c98ec8d5e186f2530e6bff1bbc431b65294c7f not found: ID does not exist" Jul 29 05:07:27.095258 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.095225 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-be8bc5-6f6c4fd6f8-l882t"] Jul 29 05:07:27.099742 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.099722 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-be8bc5-6f6c4fd6f8-l882t"] Jul 29 05:07:27.103505 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.103488 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sqmr\" (UniqueName: \"kubernetes.io/projected/353191af-c906-4fa7-b567-23bcb9d54318-kube-api-access-9sqmr\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 05:07:27.103572 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.103508 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqfhz\" (UniqueName: \"kubernetes.io/projected/90926e7a-4e34-473a-af59-d28100545dc0-kube-api-access-sqfhz\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 05:07:27.103572 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.103520 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90926e7a-4e34-473a-af59-d28100545dc0-plugins-config-volume\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 05:07:27.103572 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.103535 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353191af-c906-4fa7-b567-23bcb9d54318-plugins-config-volume\") on node \"ip-10-0-139-111.ec2.internal\" DevicePath \"\"" Jul 29 05:07:27.109666 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.109647 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-be8bc5-57d59449d8-w8h52"] Jul 29 05:07:27.114441 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.114421 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-be8bc5-57d59449d8-w8h52"] Jul 29 05:07:27.591700 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.591668 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="353191af-c906-4fa7-b567-23bcb9d54318" path="/var/lib/kubelet/pods/353191af-c906-4fa7-b567-23bcb9d54318/volumes" Jul 29 05:07:27.592043 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:27.592030 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90926e7a-4e34-473a-af59-d28100545dc0" path="/var/lib/kubelet/pods/90926e7a-4e34-473a-af59-d28100545dc0/volumes" Jul 29 05:07:34.425186 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:34.425128 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5fc9448994-dfz7l_8e9551f5-88ae-443b-84a9-49e3dbe79ed5/authorino/0.log" Jul 29 05:07:39.876114 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:39.876081 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-zxk52_47950bb6-bfdb-4449-9079-088cb903ef41/manager/0.log" Jul 29 05:07:40.152076 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:40.152003 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b9ff676b5-dgmvr_793f13a7-a5e3-4f3a-a0d1-3d5caba57f23/manager/0.log" Jul 29 05:07:40.258598 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:40.258568 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-ndfqk_0d1d3c77-b6f7-41c2-804c-cc829622aab0/server/0.log" Jul 29 05:07:40.372138 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:40.372105 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-76msd_0ea18530-6257-4356-b3c7-bd18ed2f1e6c/manager/0.log" Jul 29 05:07:40.606556 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:40.606518 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64c765c8-p2bl9_e4a99836-ced3-435f-8498-5c0ae65134cc/copy-manifests/0.log" Jul 29 05:07:40.613833 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:40.613801 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64c765c8-p2bl9_e4a99836-ced3-435f-8498-5c0ae65134cc/manager/0.log" Jul 29 05:07:41.077274 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:41.077246 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-97874b7d8-mc4q7_6500f16e-5496-4519-8b5b-354d30212cb2/maas-api/0.log" Jul 29 05:07:41.536180 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:41.536152 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-9ll76_a9918c69-9029-4a21-bbc7-86896a1c7aaa/postgres/0.log" Jul 29 05:07:42.808446 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:42.808421 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5fc9448994-dfz7l_8e9551f5-88ae-443b-84a9-49e3dbe79ed5/authorino/0.log" Jul 29 05:07:42.925275 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:42.925239 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-dsmlq_fdd47a24-06a1-4569-a22e-c58f94bed861/manager/0.log" Jul 29 05:07:43.033253 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:43.033225 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-fq588_f802ce80-3b32-4be6-bef3-7ea63ecb84f2/manager/0.log" Jul 29 05:07:43.142629 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:43.142549 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-2jgl8_709f3923-b331-49d9-9a0a-4cfec10c8137/kuadrant-console-plugin/0.log" Jul 29 05:07:43.257202 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:43.257169 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-bmsrb_f4b4a241-788d-4eef-b599-a4de941ddacb/registry-server/0.log" Jul 29 05:07:43.374127 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:43.374094 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-774gm_f724cd5b-b799-4a64-8a7b-109b83601d2b/manager/0.log" Jul 29 05:07:43.600489 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:43.600457 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-hppfm_95a2f8b7-667d-4cc9-aa41-171ed96d1d40/manager/0.log" Jul 29 05:07:43.945764 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:43.945737 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw_63fd035a-23fc-4221-a45d-19c7d74b5ae6/istio-proxy/0.log" Jul 29 05:07:44.289611 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:44.289534 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76ccb86b9-lh4m6_df92f309-917b-4659-a2a8-6e9a1baa70f9/kube-auth-proxy/0.log" Jul 29 05:07:44.403612 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:44.403576 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-kjtrs_2f96d788-589f-4bf1-8873-92b1a8ed1b1c/istio-proxy/1.log" Jul 29 05:07:44.627252 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:44.627154 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-vmjvs_8d42dd32-a9fb-49a4-9c1e-34ba3e1a96c0/payload-processing/0.log" Jul 29 05:07:45.277145 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:45.277113 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-gxwxb_e4e78827-a544-47e4-ac4b-71fb42418fd8/main/0.log" Jul 29 05:07:54.065726 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:54.065689 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-b9zpd_3fe9d7d5-24dc-4d5b-94d2-5e0fd1afcf1b/global-pull-secret-syncer/0.log" Jul 29 05:07:54.192506 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:54.192471 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-492cr_9384d535-332c-4320-a56e-a8cccbbbc5e8/konnectivity-agent/0.log" Jul 29 05:07:54.304411 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:54.304383 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-111.ec2.internal_e386db4858c90ca09ab480d4f37fcab1/haproxy/0.log" Jul 29 05:07:59.000275 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:59.000242 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5fc9448994-dfz7l_8e9551f5-88ae-443b-84a9-49e3dbe79ed5/authorino/0.log" Jul 29 05:07:59.028292 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:59.028261 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-dsmlq_fdd47a24-06a1-4569-a22e-c58f94bed861/manager/0.log" Jul 29 05:07:59.076421 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:59.076394 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-fq588_f802ce80-3b32-4be6-bef3-7ea63ecb84f2/manager/0.log" Jul 29 05:07:59.102637 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:59.102614 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-2jgl8_709f3923-b331-49d9-9a0a-4cfec10c8137/kuadrant-console-plugin/0.log" Jul 29 05:07:59.156880 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:59.156848 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-bmsrb_f4b4a241-788d-4eef-b599-a4de941ddacb/registry-server/0.log" Jul 29 05:07:59.224902 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:59.224864 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-774gm_f724cd5b-b799-4a64-8a7b-109b83601d2b/manager/0.log" Jul 29 05:07:59.330523 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:07:59.330444 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-hppfm_95a2f8b7-667d-4cc9-aa41-171ed96d1d40/manager/0.log" Jul 29 05:08:01.403434 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:01.403403 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-27rsx_64605a73-fc63-46da-ba97-768a1215cd86/node-exporter/0.log" Jul 29 05:08:01.424839 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:01.424811 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-27rsx_64605a73-fc63-46da-ba97-768a1215cd86/kube-rbac-proxy/0.log" Jul 29 05:08:01.446782 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:01.446719 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-27rsx_64605a73-fc63-46da-ba97-768a1215cd86/init-textfile/0.log" Jul 29 05:08:04.081232 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.081169 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bld4p/perf-node-gather-daemonset-qt2pk"] Jul 29 05:08:04.081820 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.081792 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c95a4a6-e48a-469e-a437-05e9c350d594" containerName="payload-processing" Jul 29 05:08:04.081820 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.081819 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c95a4a6-e48a-469e-a437-05e9c350d594" containerName="payload-processing" Jul 29 05:08:04.081964 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.081844 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="453ac86d-ebcd-47d0-97a4-5a94969abffa" containerName="payload-pre-processing" Jul 29 05:08:04.081964 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.081854 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="453ac86d-ebcd-47d0-97a4-5a94969abffa" containerName="payload-pre-processing" Jul 29 05:08:04.081964 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.081860 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b6103e8-6946-431b-b34d-5d6b63739df7" containerName="payload-processing" Jul 29 05:08:04.081964 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.081868 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b6103e8-6946-431b-b34d-5d6b63739df7" containerName="payload-processing" Jul 29 05:08:04.081964 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.081875 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90926e7a-4e34-473a-af59-d28100545dc0" containerName="payload-pre-processing" Jul 29 05:08:04.081964 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.081880 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="90926e7a-4e34-473a-af59-d28100545dc0" containerName="payload-pre-processing" Jul 29 05:08:04.081964 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.081890 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79f666dc-1e82-42b5-90fc-6e07e17af8b6" containerName="maas-api" Jul 29 05:08:04.081964 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.081898 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="79f666dc-1e82-42b5-90fc-6e07e17af8b6" containerName="maas-api" Jul 29 05:08:04.081964 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.081907 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de6fdfe2-9d65-48fc-80c3-b571655a2a4c" containerName="payload-pre-processing" Jul 29 05:08:04.081964 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.081913 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="de6fdfe2-9d65-48fc-80c3-b571655a2a4c" containerName="payload-pre-processing" Jul 29 05:08:04.081964 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.081928 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="353191af-c906-4fa7-b567-23bcb9d54318" containerName="payload-processing" Jul 29 05:08:04.081964 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.081934 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="353191af-c906-4fa7-b567-23bcb9d54318" containerName="payload-processing" Jul 29 05:08:04.081964 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.081940 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abef8972-03e1-4b93-bc01-b3649f8ea19b" containerName="payload-pre-processing" Jul 29 05:08:04.081964 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.081946 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="abef8972-03e1-4b93-bc01-b3649f8ea19b" containerName="payload-pre-processing" Jul 29 05:08:04.082643 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.082004 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c95a4a6-e48a-469e-a437-05e9c350d594" containerName="payload-processing" Jul 29 05:08:04.082643 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.082013 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b6103e8-6946-431b-b34d-5d6b63739df7" containerName="payload-processing" Jul 29 05:08:04.082643 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.082018 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="90926e7a-4e34-473a-af59-d28100545dc0" containerName="payload-pre-processing" Jul 29 05:08:04.082643 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.082023 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="de6fdfe2-9d65-48fc-80c3-b571655a2a4c" containerName="payload-pre-processing" Jul 29 05:08:04.082643 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.082029 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="abef8972-03e1-4b93-bc01-b3649f8ea19b" containerName="payload-pre-processing" Jul 29 05:08:04.082643 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.082034 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="79f666dc-1e82-42b5-90fc-6e07e17af8b6" containerName="maas-api" Jul 29 05:08:04.082643 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.082039 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="353191af-c906-4fa7-b567-23bcb9d54318" containerName="payload-processing" Jul 29 05:08:04.082643 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.082045 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="453ac86d-ebcd-47d0-97a4-5a94969abffa" containerName="payload-pre-processing" Jul 29 05:08:04.085395 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.085373 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bld4p/perf-node-gather-daemonset-qt2pk"] Jul 29 05:08:04.085509 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.085451 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bld4p/perf-node-gather-daemonset-qt2pk" Jul 29 05:08:04.087695 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.087666 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bld4p\"/\"default-dockercfg-hmmx4\"" Jul 29 05:08:04.088041 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.088022 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bld4p\"/\"openshift-service-ca.crt\"" Jul 29 05:08:04.088585 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.088569 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bld4p\"/\"kube-root-ca.crt\"" Jul 29 05:08:04.218965 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.218931 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/db17c8c7-3bf5-4f3e-a9d3-8f32986714fd-lib-modules\") pod \"perf-node-gather-daemonset-qt2pk\" (UID: \"db17c8c7-3bf5-4f3e-a9d3-8f32986714fd\") " pod="openshift-must-gather-bld4p/perf-node-gather-daemonset-qt2pk" Jul 29 05:08:04.219153 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.218970 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ksc2\" (UniqueName: \"kubernetes.io/projected/db17c8c7-3bf5-4f3e-a9d3-8f32986714fd-kube-api-access-5ksc2\") pod \"perf-node-gather-daemonset-qt2pk\" (UID: \"db17c8c7-3bf5-4f3e-a9d3-8f32986714fd\") " pod="openshift-must-gather-bld4p/perf-node-gather-daemonset-qt2pk" Jul 29 05:08:04.219153 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.219045 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db17c8c7-3bf5-4f3e-a9d3-8f32986714fd-sys\") pod \"perf-node-gather-daemonset-qt2pk\" (UID: \"db17c8c7-3bf5-4f3e-a9d3-8f32986714fd\") " pod="openshift-must-gather-bld4p/perf-node-gather-daemonset-qt2pk" Jul 29 05:08:04.219153 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.219069 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/db17c8c7-3bf5-4f3e-a9d3-8f32986714fd-proc\") pod \"perf-node-gather-daemonset-qt2pk\" (UID: \"db17c8c7-3bf5-4f3e-a9d3-8f32986714fd\") " pod="openshift-must-gather-bld4p/perf-node-gather-daemonset-qt2pk" Jul 29 05:08:04.219153 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.219092 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/db17c8c7-3bf5-4f3e-a9d3-8f32986714fd-podres\") pod \"perf-node-gather-daemonset-qt2pk\" (UID: \"db17c8c7-3bf5-4f3e-a9d3-8f32986714fd\") " pod="openshift-must-gather-bld4p/perf-node-gather-daemonset-qt2pk" Jul 29 05:08:04.319605 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.319568 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/db17c8c7-3bf5-4f3e-a9d3-8f32986714fd-lib-modules\") pod \"perf-node-gather-daemonset-qt2pk\" (UID: \"db17c8c7-3bf5-4f3e-a9d3-8f32986714fd\") " pod="openshift-must-gather-bld4p/perf-node-gather-daemonset-qt2pk" Jul 29 05:08:04.319605 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.319610 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ksc2\" (UniqueName: \"kubernetes.io/projected/db17c8c7-3bf5-4f3e-a9d3-8f32986714fd-kube-api-access-5ksc2\") pod \"perf-node-gather-daemonset-qt2pk\" (UID: \"db17c8c7-3bf5-4f3e-a9d3-8f32986714fd\") " pod="openshift-must-gather-bld4p/perf-node-gather-daemonset-qt2pk" Jul 29 05:08:04.319852 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.319748 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db17c8c7-3bf5-4f3e-a9d3-8f32986714fd-sys\") pod \"perf-node-gather-daemonset-qt2pk\" (UID: \"db17c8c7-3bf5-4f3e-a9d3-8f32986714fd\") " pod="openshift-must-gather-bld4p/perf-node-gather-daemonset-qt2pk" Jul 29 05:08:04.319852 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.319779 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/db17c8c7-3bf5-4f3e-a9d3-8f32986714fd-proc\") pod \"perf-node-gather-daemonset-qt2pk\" (UID: \"db17c8c7-3bf5-4f3e-a9d3-8f32986714fd\") " pod="openshift-must-gather-bld4p/perf-node-gather-daemonset-qt2pk" Jul 29 05:08:04.319852 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.319784 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/db17c8c7-3bf5-4f3e-a9d3-8f32986714fd-lib-modules\") pod \"perf-node-gather-daemonset-qt2pk\" (UID: \"db17c8c7-3bf5-4f3e-a9d3-8f32986714fd\") " pod="openshift-must-gather-bld4p/perf-node-gather-daemonset-qt2pk" Jul 29 05:08:04.319852 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.319796 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/db17c8c7-3bf5-4f3e-a9d3-8f32986714fd-podres\") pod \"perf-node-gather-daemonset-qt2pk\" (UID: \"db17c8c7-3bf5-4f3e-a9d3-8f32986714fd\") " pod="openshift-must-gather-bld4p/perf-node-gather-daemonset-qt2pk" Jul 29 05:08:04.319998 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.319876 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db17c8c7-3bf5-4f3e-a9d3-8f32986714fd-sys\") pod \"perf-node-gather-daemonset-qt2pk\" (UID: \"db17c8c7-3bf5-4f3e-a9d3-8f32986714fd\") " pod="openshift-must-gather-bld4p/perf-node-gather-daemonset-qt2pk" Jul 29 05:08:04.319998 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.319934 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/db17c8c7-3bf5-4f3e-a9d3-8f32986714fd-proc\") pod \"perf-node-gather-daemonset-qt2pk\" (UID: \"db17c8c7-3bf5-4f3e-a9d3-8f32986714fd\") " pod="openshift-must-gather-bld4p/perf-node-gather-daemonset-qt2pk" Jul 29 05:08:04.319998 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.319971 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/db17c8c7-3bf5-4f3e-a9d3-8f32986714fd-podres\") pod \"perf-node-gather-daemonset-qt2pk\" (UID: \"db17c8c7-3bf5-4f3e-a9d3-8f32986714fd\") " pod="openshift-must-gather-bld4p/perf-node-gather-daemonset-qt2pk" Jul 29 05:08:04.328474 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.328444 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ksc2\" (UniqueName: \"kubernetes.io/projected/db17c8c7-3bf5-4f3e-a9d3-8f32986714fd-kube-api-access-5ksc2\") pod \"perf-node-gather-daemonset-qt2pk\" (UID: \"db17c8c7-3bf5-4f3e-a9d3-8f32986714fd\") " pod="openshift-must-gather-bld4p/perf-node-gather-daemonset-qt2pk" Jul 29 05:08:04.398827 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.398743 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bld4p/perf-node-gather-daemonset-qt2pk" Jul 29 05:08:04.536177 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:04.536148 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bld4p/perf-node-gather-daemonset-qt2pk"] Jul 29 05:08:04.538501 ip-10-0-139-111 kubenswrapper[2579]: W0729 05:08:04.538475 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddb17c8c7_3bf5_4f3e_a9d3_8f32986714fd.slice/crio-39fe122ef8b8fdeeb8ad98a4e61c149d2296f12b539f86a4ac335cc3661619a5 WatchSource:0}: Error finding container 39fe122ef8b8fdeeb8ad98a4e61c149d2296f12b539f86a4ac335cc3661619a5: Status 404 returned error can't find the container with id 39fe122ef8b8fdeeb8ad98a4e61c149d2296f12b539f86a4ac335cc3661619a5 Jul 29 05:08:05.259098 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:05.259062 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bld4p/perf-node-gather-daemonset-qt2pk" event={"ID":"db17c8c7-3bf5-4f3e-a9d3-8f32986714fd","Type":"ContainerStarted","Data":"e01c82db830f08cbf9798dac50c4b68b25405c449a37be790f2c7b58646c8ed1"} Jul 29 05:08:05.259559 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:05.259115 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bld4p/perf-node-gather-daemonset-qt2pk" event={"ID":"db17c8c7-3bf5-4f3e-a9d3-8f32986714fd","Type":"ContainerStarted","Data":"39fe122ef8b8fdeeb8ad98a4e61c149d2296f12b539f86a4ac335cc3661619a5"} Jul 29 05:08:05.259559 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:05.259176 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bld4p/perf-node-gather-daemonset-qt2pk" Jul 29 05:08:05.273753 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:05.273704 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bld4p/perf-node-gather-daemonset-qt2pk" podStartSLOduration=1.2736897950000001 podStartE2EDuration="1.273689795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 05:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 05:08:05.273534209 +0000 UTC m=+3866.132908934" watchObservedRunningTime="2026-07-29 05:08:05.273689795 +0000 UTC m=+3866.133064519" Jul 29 05:08:06.659970 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:06.659940 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lfjbp_ea98d97f-f580-434b-8d20-b687c0d951fd/dns/0.log" Jul 29 05:08:06.684243 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:06.684202 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lfjbp_ea98d97f-f580-434b-8d20-b687c0d951fd/kube-rbac-proxy/0.log" Jul 29 05:08:06.756285 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:06.756250 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v8d4n_dc47e5bf-dc39-4b81-ba67-5c00203489e0/dns-node-resolver/0.log" Jul 29 05:08:07.288850 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:07.288822 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-bbfb64c88-lv4w2_3f243811-a083-4a5d-9043-47b372431bce/registry/0.log" Jul 29 05:08:07.360580 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:07.360554 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wwg6z_0055fae1-bd7e-4e87-834c-3a317313ae0c/node-ca/0.log" Jul 29 05:08:08.294759 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:08.294727 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw_63fd035a-23fc-4221-a45d-19c7d74b5ae6/istio-proxy/0.log" Jul 29 05:08:08.552092 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:08.552015 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76ccb86b9-lh4m6_df92f309-917b-4659-a2a8-6e9a1baa70f9/kube-auth-proxy/0.log" Jul 29 05:08:08.583929 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:08.583900 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-kjtrs_2f96d788-589f-4bf1-8873-92b1a8ed1b1c/istio-proxy/0.log" Jul 29 05:08:08.608834 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:08.608806 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-kjtrs_2f96d788-589f-4bf1-8873-92b1a8ed1b1c/istio-proxy/1.log" Jul 29 05:08:08.668619 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:08.668589 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-vmjvs_8d42dd32-a9fb-49a4-9c1e-34ba3e1a96c0/payload-processing/0.log" Jul 29 05:08:09.526171 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:09.526138 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bprlw_dcdad901-fa84-42eb-8e6d-70c86d2639cb/serve-healthcheck-canary/0.log" Jul 29 05:08:10.097822 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:10.097794 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-56kwm_1deee91e-bb4a-4511-953d-9fd13697bdc1/kube-rbac-proxy/0.log" Jul 29 05:08:10.117318 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:10.117293 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-56kwm_1deee91e-bb4a-4511-953d-9fd13697bdc1/exporter/0.log" Jul 29 05:08:10.136926 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:10.136898 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-56kwm_1deee91e-bb4a-4511-953d-9fd13697bdc1/extractor/0.log" Jul 29 05:08:11.273349 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:11.273322 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bld4p/perf-node-gather-daemonset-qt2pk" Jul 29 05:08:12.257112 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:12.257081 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-zxk52_47950bb6-bfdb-4449-9079-088cb903ef41/manager/0.log" Jul 29 05:08:12.823802 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:12.823765 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b9ff676b5-dgmvr_793f13a7-a5e3-4f3a-a0d1-3d5caba57f23/manager/0.log" Jul 29 05:08:12.845545 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:12.845521 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-ndfqk_0d1d3c77-b6f7-41c2-804c-cc829622aab0/server/0.log" Jul 29 05:08:12.918652 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:12.918627 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-76msd_0ea18530-6257-4356-b3c7-bd18ed2f1e6c/manager/0.log" Jul 29 05:08:13.077375 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:13.077328 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64c765c8-p2bl9_e4a99836-ced3-435f-8498-5c0ae65134cc/manager/0.log" Jul 29 05:08:13.099180 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:13.099151 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64c765c8-p2bl9_e4a99836-ced3-435f-8498-5c0ae65134cc/copy-manifests/0.log" Jul 29 05:08:14.339985 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:14.339954 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw_63fd035a-23fc-4221-a45d-19c7d74b5ae6/istio-proxy/0.log" Jul 29 05:08:14.547974 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:14.547947 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76ccb86b9-lh4m6_df92f309-917b-4659-a2a8-6e9a1baa70f9/kube-auth-proxy/0.log" Jul 29 05:08:14.568053 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:14.567995 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-kjtrs_2f96d788-589f-4bf1-8873-92b1a8ed1b1c/istio-proxy/0.log" Jul 29 05:08:14.594125 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:14.594029 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-kjtrs_2f96d788-589f-4bf1-8873-92b1a8ed1b1c/istio-proxy/1.log" Jul 29 05:08:14.636985 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:14.636961 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-vmjvs_8d42dd32-a9fb-49a4-9c1e-34ba3e1a96c0/payload-processing/0.log" Jul 29 05:08:16.106622 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:16.106592 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-qgxqw_63fd035a-23fc-4221-a45d-19c7d74b5ae6/istio-proxy/0.log" Jul 29 05:08:16.311650 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:16.311622 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76ccb86b9-lh4m6_df92f309-917b-4659-a2a8-6e9a1baa70f9/kube-auth-proxy/0.log" Jul 29 05:08:16.334941 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:16.334916 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-kjtrs_2f96d788-589f-4bf1-8873-92b1a8ed1b1c/istio-proxy/0.log" Jul 29 05:08:16.361170 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:16.361105 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-kjtrs_2f96d788-589f-4bf1-8873-92b1a8ed1b1c/istio-proxy/1.log" Jul 29 05:08:16.410451 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:16.410422 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-vmjvs_8d42dd32-a9fb-49a4-9c1e-34ba3e1a96c0/payload-processing/0.log" Jul 29 05:08:19.448710 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:19.448683 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-c8mlp_2dafc4cf-b1a5-4907-8f24-a30ca4f08b30/migrator/0.log" Jul 29 05:08:19.474120 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:19.474094 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-c8mlp_2dafc4cf-b1a5-4907-8f24-a30ca4f08b30/graceful-termination/0.log" Jul 29 05:08:21.193094 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:21.193063 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zjpnt_a4d18741-fe36-4f1a-a290-795bfc955fb0/kube-multus-additional-cni-plugins/0.log" Jul 29 05:08:21.214201 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:21.214177 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zjpnt_a4d18741-fe36-4f1a-a290-795bfc955fb0/egress-router-binary-copy/0.log" Jul 29 05:08:21.236766 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:21.236736 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zjpnt_a4d18741-fe36-4f1a-a290-795bfc955fb0/cni-plugins/0.log" Jul 29 05:08:21.257988 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:21.257963 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zjpnt_a4d18741-fe36-4f1a-a290-795bfc955fb0/bond-cni-plugin/0.log" Jul 29 05:08:21.280317 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:21.280288 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zjpnt_a4d18741-fe36-4f1a-a290-795bfc955fb0/routeoverride-cni/0.log" Jul 29 05:08:21.301549 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:21.301523 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zjpnt_a4d18741-fe36-4f1a-a290-795bfc955fb0/whereabouts-cni-bincopy/0.log" Jul 29 05:08:21.322931 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:21.322907 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zjpnt_a4d18741-fe36-4f1a-a290-795bfc955fb0/whereabouts-cni/0.log" Jul 29 05:08:21.446903 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:21.446827 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-snqz4_ed77db79-64ab-4dda-8b0c-3ab2ec321512/kube-multus/0.log" Jul 29 05:08:21.559546 ip-10-0-139-111 kubenswrapper[2579]: I0729 05:08:21.559512 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zbrk8_9fa6e244-be92-4245-a54e-7aa303802f4b/network-metrics-daemon/0.log"