Sep 09 10:24:08.333467 ip-10-0-139-133 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 09 10:24:08.333477 ip-10-0-139-133 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 09 10:24:08.333484 ip-10-0-139-133 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 09 10:24:08.333633 ip-10-0-139-133 systemd[1]: Failed to start Kubernetes Kubelet. Sep 09 10:24:19.678180 ip-10-0-139-133 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 09 10:24:19.678198 ip-10-0-139-133 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 87f5b2441336488ea7f128a6cf245100 -- Sep 09 10:25:59.710832 ip-10-0-139-133 systemd[1]: Starting Kubernetes Kubelet... Sep 09 10:26:00.110883 ip-10-0-139-133 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 10:26:00.110883 ip-10-0-139-133 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 09 10:26:00.110883 ip-10-0-139-133 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 10:26:00.110883 ip-10-0-139-133 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 09 10:26:00.110883 ip-10-0-139-133 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 10:26:00.113714 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.113629 2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 09 10:26:00.118079 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118063 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 10:26:00.118079 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118079 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 10:26:00.118150 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118082 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 10:26:00.118150 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118086 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 10:26:00.118150 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118089 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 10:26:00.118150 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118092 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 10:26:00.118150 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118096 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 10:26:00.118150 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118100 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 10:26:00.118150 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118103 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 10:26:00.118150 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118106 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 10:26:00.118150 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118109 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 10:26:00.118150 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118114 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 10:26:00.118150 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118118 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 10:26:00.118150 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118129 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 10:26:00.118150 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118133 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 10:26:00.118150 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118136 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 10:26:00.118150 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118139 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 10:26:00.118150 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118142 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 10:26:00.118150 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118144 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 10:26:00.118150 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118147 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 10:26:00.118150 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118151 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 10:26:00.118747 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118154 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 10:26:00.118747 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118157 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 10:26:00.118747 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118160 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 10:26:00.118747 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118163 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 10:26:00.118747 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118166 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 10:26:00.118747 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118168 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 10:26:00.118747 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118171 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 10:26:00.118747 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118173 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 10:26:00.118747 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118176 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 10:26:00.118747 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118179 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 10:26:00.118747 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118181 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 10:26:00.118747 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118184 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 10:26:00.118747 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118187 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 10:26:00.118747 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118189 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 10:26:00.118747 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118191 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 10:26:00.118747 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118194 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 10:26:00.118747 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118196 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 10:26:00.118747 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118199 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 10:26:00.118747 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118201 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 10:26:00.118747 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118204 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 10:26:00.119431 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118206 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 10:26:00.119431 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118209 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 10:26:00.119431 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118211 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 10:26:00.119431 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118214 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 10:26:00.119431 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118217 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 10:26:00.119431 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118259 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 10:26:00.119431 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118308 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 10:26:00.119431 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118314 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 10:26:00.119431 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118318 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 10:26:00.119431 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118321 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 10:26:00.119431 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118325 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 09 10:26:00.119431 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118327 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 10:26:00.119431 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118330 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 10:26:00.119431 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118333 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 10:26:00.119431 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118336 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 10:26:00.119431 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118339 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 10:26:00.119431 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118343 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 10:26:00.119431 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118347 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 10:26:00.119431 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118354 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 10:26:00.119431 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118356 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 10:26:00.119938 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118360 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 10:26:00.119938 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118364 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 10:26:00.119938 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118369 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 10:26:00.119938 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118373 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 10:26:00.119938 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118377 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 10:26:00.119938 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118382 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 10:26:00.119938 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118386 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 10:26:00.119938 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118461 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 10:26:00.119938 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118478 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 10:26:00.119938 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118485 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 10:26:00.119938 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118490 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 10:26:00.119938 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118509 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 10:26:00.119938 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118516 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 10:26:00.119938 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118521 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 10:26:00.119938 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118526 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 10:26:00.119938 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118530 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 10:26:00.119938 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118535 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 10:26:00.119938 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118540 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 10:26:00.119938 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118545 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 10:26:00.119938 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118554 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 10:26:00.120422 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118560 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 10:26:00.120422 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118564 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 10:26:00.120422 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118569 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 10:26:00.120422 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118573 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 10:26:00.120422 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118577 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 10:26:00.120422 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118582 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 10:26:00.120422 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118586 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 10:26:00.120422 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118590 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 10:26:00.120422 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118595 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 10:26:00.120422 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118603 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 10:26:00.120422 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118607 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 10:26:00.120422 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118616 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 10:26:00.120422 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118620 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 10:26:00.120422 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118624 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 10:26:00.120422 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118628 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 10:26:00.120422 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118632 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 10:26:00.120422 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118637 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 10:26:00.120422 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.118643 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 10:26:00.120422 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120034 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 10:26:00.120422 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120044 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 10:26:00.120962 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120048 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 10:26:00.120962 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120051 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 10:26:00.120962 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120054 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 10:26:00.120962 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120057 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 10:26:00.120962 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120060 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 10:26:00.120962 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120063 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 10:26:00.120962 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120065 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 10:26:00.120962 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120068 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 10:26:00.120962 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120071 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 10:26:00.120962 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120073 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 10:26:00.120962 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120076 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 10:26:00.120962 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120079 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 10:26:00.120962 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120081 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 10:26:00.120962 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120086 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 10:26:00.120962 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120090 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 10:26:00.120962 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120093 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 10:26:00.120962 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120099 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 10:26:00.120962 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120102 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 10:26:00.120962 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120105 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 10:26:00.120962 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120108 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 10:26:00.121466 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120110 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 10:26:00.121466 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120113 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 10:26:00.121466 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120116 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 10:26:00.121466 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120118 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 10:26:00.121466 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120121 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 10:26:00.121466 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120123 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 10:26:00.121466 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120126 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 10:26:00.121466 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120128 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 10:26:00.121466 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120131 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 10:26:00.121466 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120133 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 10:26:00.121466 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120136 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 10:26:00.121466 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120140 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 10:26:00.121466 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120143 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 10:26:00.121466 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120145 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 10:26:00.121466 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120148 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 10:26:00.121466 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120150 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 10:26:00.121466 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120153 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 10:26:00.121466 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120155 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 10:26:00.121466 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120158 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 09 10:26:00.121466 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120160 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 10:26:00.121984 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120163 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 10:26:00.121984 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120165 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 10:26:00.121984 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120168 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 10:26:00.121984 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120170 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 10:26:00.121984 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120173 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 10:26:00.121984 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120175 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 10:26:00.121984 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120179 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 10:26:00.121984 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120182 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 10:26:00.121984 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120186 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 10:26:00.121984 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120190 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 10:26:00.121984 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120193 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 10:26:00.121984 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120195 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 10:26:00.121984 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120198 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 10:26:00.121984 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120201 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 10:26:00.121984 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120204 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 10:26:00.121984 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120207 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 10:26:00.121984 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120209 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 10:26:00.121984 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120212 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 10:26:00.121984 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120214 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 10:26:00.122458 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120216 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 10:26:00.122458 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120219 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 10:26:00.122458 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120227 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 10:26:00.122458 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120235 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 10:26:00.122458 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120239 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 10:26:00.122458 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120241 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 10:26:00.122458 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120244 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 10:26:00.122458 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120246 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 10:26:00.122458 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120249 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 10:26:00.122458 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120252 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 10:26:00.122458 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120255 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 10:26:00.122458 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120257 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 10:26:00.122458 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120260 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 10:26:00.122458 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120263 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 10:26:00.122458 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120265 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 10:26:00.122458 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120268 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 10:26:00.122458 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120270 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 10:26:00.122458 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120273 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 10:26:00.122458 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120275 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 10:26:00.122458 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120278 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 10:26:00.122971 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120280 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 10:26:00.122971 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120284 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 10:26:00.122971 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120286 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 10:26:00.122971 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120289 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 10:26:00.122971 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120292 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 10:26:00.122971 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120295 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 10:26:00.122971 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120298 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 10:26:00.122971 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120301 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 10:26:00.122971 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120304 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 10:26:00.122971 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120307 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 10:26:00.122971 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120311 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 10:26:00.122971 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120314 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 10:26:00.122971 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120316 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 10:26:00.122971 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120318 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 10:26:00.122971 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120321 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 10:26:00.122971 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120325 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 10:26:00.122971 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120328 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 10:26:00.122971 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120331 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 10:26:00.122971 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120753 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 10:26:00.123440 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120759 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 10:26:00.123440 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120763 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 10:26:00.123440 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120767 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 10:26:00.123440 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120770 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 10:26:00.123440 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120773 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 10:26:00.123440 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120777 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 10:26:00.123440 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120781 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 10:26:00.123440 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120784 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 10:26:00.123440 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120787 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 10:26:00.123440 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120790 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 10:26:00.123440 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120792 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 10:26:00.123440 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120795 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 10:26:00.123440 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120798 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 10:26:00.123440 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120802 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 10:26:00.123440 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120805 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 09 10:26:00.123440 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120807 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 10:26:00.123440 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120810 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 10:26:00.123440 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120813 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 10:26:00.123440 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120815 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 10:26:00.123918 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120818 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 10:26:00.123918 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120821 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 10:26:00.123918 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120823 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 10:26:00.123918 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120825 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 10:26:00.123918 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120828 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 10:26:00.123918 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120830 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 10:26:00.123918 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120833 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 10:26:00.123918 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120835 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 10:26:00.123918 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120838 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 10:26:00.123918 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120846 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 10:26:00.123918 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120850 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 10:26:00.123918 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120852 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 10:26:00.123918 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120855 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 10:26:00.123918 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120857 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 10:26:00.123918 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120860 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 10:26:00.123918 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120862 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 10:26:00.123918 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120865 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 10:26:00.123918 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120867 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 10:26:00.123918 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120870 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 10:26:00.123918 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120872 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 10:26:00.124493 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120875 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 10:26:00.124493 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120878 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 10:26:00.124493 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120881 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 10:26:00.124493 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120883 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 10:26:00.124493 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120886 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 10:26:00.124493 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120888 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 10:26:00.124493 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120891 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 10:26:00.124493 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120894 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 10:26:00.124493 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120896 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 10:26:00.124493 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120899 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 10:26:00.124493 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120902 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 10:26:00.124493 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120904 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 10:26:00.124493 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120907 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 10:26:00.124493 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120909 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 10:26:00.124493 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120912 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 10:26:00.124493 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120914 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 10:26:00.124493 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120917 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 10:26:00.124493 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120919 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 10:26:00.124493 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120922 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 10:26:00.124493 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120924 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 10:26:00.125001 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120927 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 10:26:00.125001 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120930 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 10:26:00.125001 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120934 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 10:26:00.125001 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120936 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 10:26:00.125001 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120939 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 10:26:00.125001 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120941 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 10:26:00.125001 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120944 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 10:26:00.125001 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120946 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 10:26:00.125001 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120949 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 10:26:00.125001 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120952 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 10:26:00.125001 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120954 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 10:26:00.125001 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120957 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 10:26:00.125001 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120959 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 10:26:00.125001 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120962 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 10:26:00.125001 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120964 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 10:26:00.125001 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120967 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 10:26:00.125001 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120969 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 10:26:00.125001 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120972 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 10:26:00.125001 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120975 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 10:26:00.125001 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120978 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 10:26:00.125488 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120981 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 10:26:00.125488 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120984 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 10:26:00.125488 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120986 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 10:26:00.125488 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120989 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 10:26:00.125488 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120991 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 10:26:00.125488 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120993 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 10:26:00.125488 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.120996 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 10:26:00.125488 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121000 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 10:26:00.125488 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121003 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 10:26:00.125488 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121006 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 10:26:00.125488 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121009 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 10:26:00.125488 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121012 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 10:26:00.125488 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121014 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 10:26:00.125488 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121017 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 10:26:00.125488 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121020 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 10:26:00.125488 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121024 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 10:26:00.125488 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121026 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 10:26:00.125488 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121029 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 10:26:00.125488 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121032 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 10:26:00.126005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121107 2559 flags.go:64] FLAG: --address="0.0.0.0" Sep 09 10:26:00.126005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121145 2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 09 10:26:00.126005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121151 2559 flags.go:64] FLAG: --anonymous-auth="true" Sep 09 10:26:00.126005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121156 2559 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 09 10:26:00.126005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121160 2559 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 09 10:26:00.126005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121163 2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 09 10:26:00.126005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121167 2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 09 10:26:00.126005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121171 2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 09 10:26:00.126005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121175 2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 09 10:26:00.126005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121177 2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 09 10:26:00.126005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121181 2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 09 10:26:00.126005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121184 2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 09 10:26:00.126005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121187 2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 09 10:26:00.126005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121190 2559 flags.go:64] FLAG: --cgroup-root="" Sep 09 10:26:00.126005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121193 2559 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 09 10:26:00.126005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121196 2559 flags.go:64] FLAG: --client-ca-file="" Sep 09 10:26:00.126005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121199 2559 flags.go:64] FLAG: --cloud-provider="external" Sep 09 10:26:00.126005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121202 2559 flags.go:64] FLAG: --cluster-dns="[]" Sep 09 10:26:00.126005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121206 2559 flags.go:64] FLAG: --cluster-domain="" Sep 09 10:26:00.126005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121208 2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 09 10:26:00.126005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121211 2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 09 10:26:00.126005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121214 2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 09 10:26:00.126005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121218 2559 flags.go:64] FLAG: --container-log-max-files="5" Sep 09 10:26:00.126005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121222 2559 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 09 10:26:00.126585 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121224 2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 09 10:26:00.126585 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121227 2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 09 10:26:00.126585 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121230 2559 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 09 10:26:00.126585 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121234 2559 flags.go:64] FLAG: --contention-profiling="false" Sep 09 10:26:00.126585 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121237 2559 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 09 10:26:00.126585 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121241 2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 09 10:26:00.126585 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121244 2559 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 09 10:26:00.126585 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121247 2559 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 09 10:26:00.126585 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121251 2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 09 10:26:00.126585 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121254 2559 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 09 10:26:00.126585 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121259 2559 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 09 10:26:00.126585 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121262 2559 flags.go:64] FLAG: --enable-load-reader="false" Sep 09 10:26:00.126585 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121264 2559 flags.go:64] FLAG: --enable-server="true" Sep 09 10:26:00.126585 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121267 2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 09 10:26:00.126585 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121272 2559 flags.go:64] FLAG: --event-burst="100" Sep 09 10:26:00.126585 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121275 2559 flags.go:64] FLAG: --event-qps="50" Sep 09 10:26:00.126585 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121278 2559 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 09 10:26:00.126585 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121281 2559 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 09 10:26:00.126585 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121284 2559 flags.go:64] FLAG: --eviction-hard="" Sep 09 10:26:00.126585 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121287 2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 09 10:26:00.126585 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121290 2559 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 09 10:26:00.126585 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121293 2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 09 10:26:00.126585 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121296 2559 flags.go:64] FLAG: --eviction-soft="" Sep 09 10:26:00.126585 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121299 2559 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 09 10:26:00.126585 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121302 2559 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 09 10:26:00.127241 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121304 2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 09 10:26:00.127241 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121307 2559 flags.go:64] FLAG: --experimental-mounter-path="" Sep 09 10:26:00.127241 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121310 2559 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 09 10:26:00.127241 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121314 2559 flags.go:64] FLAG: --fail-swap-on="true" Sep 09 10:26:00.127241 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121316 2559 flags.go:64] FLAG: --feature-gates="" Sep 09 10:26:00.127241 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121320 2559 flags.go:64] FLAG: --file-check-frequency="20s" Sep 09 10:26:00.127241 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121323 2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 09 10:26:00.127241 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121326 2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 09 10:26:00.127241 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121329 2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 09 10:26:00.127241 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121331 2559 flags.go:64] FLAG: --healthz-port="10248" Sep 09 10:26:00.127241 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121335 2559 flags.go:64] FLAG: --help="false" Sep 09 10:26:00.127241 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121338 2559 flags.go:64] FLAG: --hostname-override="ip-10-0-139-133.ec2.internal" Sep 09 10:26:00.127241 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121340 2559 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 09 10:26:00.127241 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121344 2559 flags.go:64] FLAG: --http-check-frequency="20s" Sep 09 10:26:00.127241 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121347 2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 09 10:26:00.127241 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121350 2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 09 10:26:00.127241 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121353 2559 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 09 10:26:00.127241 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121358 2559 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 09 10:26:00.127241 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121361 2559 flags.go:64] FLAG: --image-service-endpoint="" Sep 09 10:26:00.127241 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121363 2559 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 09 10:26:00.127241 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121366 2559 flags.go:64] FLAG: --kube-api-burst="100" Sep 09 10:26:00.127241 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121369 2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 09 10:26:00.127241 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121372 2559 flags.go:64] FLAG: --kube-api-qps="50" Sep 09 10:26:00.127241 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121375 2559 flags.go:64] FLAG: --kube-reserved="" Sep 09 10:26:00.127861 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121377 2559 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 09 10:26:00.127861 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121380 2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 09 10:26:00.127861 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121383 2559 flags.go:64] FLAG: --kubelet-cgroups="" Sep 09 10:26:00.127861 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121386 2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 09 10:26:00.127861 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121389 2559 flags.go:64] FLAG: --lock-file="" Sep 09 10:26:00.127861 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121391 2559 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 09 10:26:00.127861 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121394 2559 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 09 10:26:00.127861 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121397 2559 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 09 10:26:00.127861 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121402 2559 flags.go:64] FLAG: --log-json-split-stream="false" Sep 09 10:26:00.127861 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121405 2559 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 09 10:26:00.127861 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121408 2559 flags.go:64] FLAG: --log-text-split-stream="false" Sep 09 10:26:00.127861 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121411 2559 flags.go:64] FLAG: --logging-format="text" Sep 09 10:26:00.127861 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121414 2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 09 10:26:00.127861 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121417 2559 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 09 10:26:00.127861 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121420 2559 flags.go:64] FLAG: --manifest-url="" Sep 09 10:26:00.127861 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121424 2559 flags.go:64] FLAG: --manifest-url-header="" Sep 09 10:26:00.127861 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121428 2559 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 09 10:26:00.127861 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121431 2559 flags.go:64] FLAG: --max-open-files="1000000" Sep 09 10:26:00.127861 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121435 2559 flags.go:64] FLAG: --max-pods="110" Sep 09 10:26:00.127861 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121437 2559 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 09 10:26:00.127861 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121440 2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 09 10:26:00.127861 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121443 2559 flags.go:64] FLAG: --memory-manager-policy="None" Sep 09 10:26:00.127861 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121447 2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 09 10:26:00.127861 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121450 2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 09 10:26:00.127861 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121453 2559 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 09 10:26:00.128482 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121458 2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 09 10:26:00.128482 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121464 2559 flags.go:64] FLAG: --node-status-max-images="50" Sep 09 10:26:00.128482 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121468 2559 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 09 10:26:00.128482 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121471 2559 flags.go:64] FLAG: --oom-score-adj="-999" Sep 09 10:26:00.128482 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121474 2559 flags.go:64] FLAG: --pod-cidr="" Sep 09 10:26:00.128482 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121477 2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 09 10:26:00.128482 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121482 2559 flags.go:64] FLAG: --pod-manifest-path="" Sep 09 10:26:00.128482 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121485 2559 flags.go:64] FLAG: --pod-max-pids="-1" Sep 09 10:26:00.128482 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121488 2559 flags.go:64] FLAG: --pods-per-core="0" Sep 09 10:26:00.128482 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121491 2559 flags.go:64] FLAG: --port="10250" Sep 09 10:26:00.128482 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121494 2559 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 09 10:26:00.128482 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121509 2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b1a53b94b920c718" Sep 09 10:26:00.128482 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121513 2559 flags.go:64] FLAG: --qos-reserved="" Sep 09 10:26:00.128482 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121516 2559 flags.go:64] FLAG: --read-only-port="10255" Sep 09 10:26:00.128482 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121519 2559 flags.go:64] FLAG: --register-node="true" Sep 09 10:26:00.128482 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121522 2559 flags.go:64] FLAG: --register-with-taints="" Sep 09 10:26:00.128482 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121525 2559 flags.go:64] FLAG: --registry-burst="10" Sep 09 10:26:00.128482 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121528 2559 flags.go:64] FLAG: --registry-qps="5" Sep 09 10:26:00.128482 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121531 2559 flags.go:64] FLAG: --reserved-cpus="" Sep 09 10:26:00.128482 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121534 2559 flags.go:64] FLAG: --reserved-memory="" Sep 09 10:26:00.128482 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121538 2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 09 10:26:00.128482 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121541 2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 09 10:26:00.128482 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121543 2559 flags.go:64] FLAG: --rotate-certificates="false" Sep 09 10:26:00.128482 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121548 2559 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 09 10:26:00.128482 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121551 2559 flags.go:64] FLAG: --runonce="false" Sep 09 10:26:00.129112 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121554 2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 09 10:26:00.129112 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121557 2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 09 10:26:00.129112 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121560 2559 flags.go:64] FLAG: --seccomp-default="false" Sep 09 10:26:00.129112 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121562 2559 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 09 10:26:00.129112 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121565 2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 09 10:26:00.129112 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121569 2559 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 09 10:26:00.129112 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121572 2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 09 10:26:00.129112 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121575 2559 flags.go:64] FLAG: --storage-driver-password="root" Sep 09 10:26:00.129112 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121578 2559 flags.go:64] FLAG: --storage-driver-secure="false" Sep 09 10:26:00.129112 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121581 2559 flags.go:64] FLAG: --storage-driver-table="stats" Sep 09 10:26:00.129112 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121584 2559 flags.go:64] FLAG: --storage-driver-user="root" Sep 09 10:26:00.129112 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121587 2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 09 10:26:00.129112 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121590 2559 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 09 10:26:00.129112 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121593 2559 flags.go:64] FLAG: --system-cgroups="" Sep 09 10:26:00.129112 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121595 2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 09 10:26:00.129112 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121601 2559 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 09 10:26:00.129112 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121603 2559 flags.go:64] FLAG: --tls-cert-file="" Sep 09 10:26:00.129112 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121606 2559 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 09 10:26:00.129112 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121610 2559 flags.go:64] FLAG: --tls-min-version="" Sep 09 10:26:00.129112 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121612 2559 flags.go:64] FLAG: --tls-private-key-file="" Sep 09 10:26:00.129112 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121615 2559 flags.go:64] FLAG: --topology-manager-policy="none" Sep 09 10:26:00.129112 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121618 2559 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 09 10:26:00.129112 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121620 2559 flags.go:64] FLAG: --topology-manager-scope="container" Sep 09 10:26:00.129112 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121623 2559 flags.go:64] FLAG: --v="2" Sep 09 10:26:00.129112 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121627 2559 flags.go:64] FLAG: --version="false" Sep 09 10:26:00.129772 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121631 2559 flags.go:64] FLAG: --vmodule="" Sep 09 10:26:00.129772 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121634 2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 09 10:26:00.129772 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.121637 2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 09 10:26:00.129772 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121753 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 10:26:00.129772 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121757 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 10:26:00.129772 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121763 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 10:26:00.129772 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121766 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 10:26:00.129772 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121768 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 10:26:00.129772 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121771 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 10:26:00.129772 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121773 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 09 10:26:00.129772 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121776 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 10:26:00.129772 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121779 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 10:26:00.129772 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121781 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 10:26:00.129772 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121785 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 10:26:00.129772 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121788 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 10:26:00.129772 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121791 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 10:26:00.129772 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121794 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 10:26:00.129772 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121796 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 10:26:00.129772 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121799 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 10:26:00.129772 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121802 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 10:26:00.129772 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121804 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 10:26:00.130289 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121807 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 10:26:00.130289 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121809 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 10:26:00.130289 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121812 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 10:26:00.130289 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121814 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 10:26:00.130289 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121817 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 10:26:00.130289 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121819 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 10:26:00.130289 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121822 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 10:26:00.130289 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121824 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 10:26:00.130289 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121827 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 10:26:00.130289 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121830 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 10:26:00.130289 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121834 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 10:26:00.130289 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121837 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 10:26:00.130289 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121840 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 10:26:00.130289 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121843 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 10:26:00.130289 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121846 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 10:26:00.130289 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121849 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 10:26:00.130289 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121853 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 10:26:00.130289 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121864 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 10:26:00.130289 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121868 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 10:26:00.130764 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121871 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 10:26:00.130764 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121874 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 10:26:00.130764 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121877 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 10:26:00.130764 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121880 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 10:26:00.130764 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121882 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 10:26:00.130764 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121885 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 10:26:00.130764 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121889 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 10:26:00.130764 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121891 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 10:26:00.130764 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121894 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 10:26:00.130764 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121897 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 10:26:00.130764 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121900 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 10:26:00.130764 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121903 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 10:26:00.130764 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121905 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 10:26:00.130764 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121908 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 10:26:00.130764 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121910 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 10:26:00.130764 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121913 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 10:26:00.130764 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121915 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 10:26:00.130764 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121918 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 10:26:00.130764 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121920 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 10:26:00.130764 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121923 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 10:26:00.131259 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121925 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 10:26:00.131259 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121928 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 10:26:00.131259 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121930 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 10:26:00.131259 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121933 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 10:26:00.131259 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121936 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 10:26:00.131259 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121938 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 10:26:00.131259 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121941 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 10:26:00.131259 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121943 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 10:26:00.131259 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121946 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 10:26:00.131259 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121950 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 10:26:00.131259 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121953 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 10:26:00.131259 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121955 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 10:26:00.131259 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121958 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 10:26:00.131259 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121960 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 10:26:00.131259 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121963 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 10:26:00.131259 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121965 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 10:26:00.131259 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121968 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 10:26:00.131259 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121971 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 10:26:00.131259 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121974 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 10:26:00.131259 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121978 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 10:26:00.131770 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121980 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 10:26:00.131770 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121983 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 10:26:00.131770 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121985 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 10:26:00.131770 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121988 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 10:26:00.131770 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121991 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 10:26:00.131770 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121994 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 10:26:00.131770 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121997 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 10:26:00.131770 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.121999 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 10:26:00.131770 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.122002 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 10:26:00.131770 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.122004 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 10:26:00.131770 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.122007 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 10:26:00.131770 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.122009 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 10:26:00.131770 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.122012 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 10:26:00.131770 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.122014 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 10:26:00.131770 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.122017 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 10:26:00.131770 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.122021 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 10:26:00.131770 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.122024 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 10:26:00.131770 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.122028 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 10:26:00.131770 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.122031 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 10:26:00.132229 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.122034 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 10:26:00.132229 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.122037 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 10:26:00.132229 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.122041 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 10:26:00.132229 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.122658 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 10:26:00.132229 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.129320 2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 09 10:26:00.132229 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.129337 2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 09 10:26:00.132229 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.130077 2559 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 09 10:26:00.132229 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.130085 2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 09 10:26:00.132229 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130145 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 10:26:00.132229 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130151 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 10:26:00.132229 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130155 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 10:26:00.132229 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130158 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 10:26:00.132229 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130162 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 10:26:00.132229 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130165 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 10:26:00.132630 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130168 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 10:26:00.132630 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130171 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 10:26:00.132630 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130173 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 10:26:00.132630 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130176 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 10:26:00.132630 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130178 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 10:26:00.132630 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130181 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 10:26:00.132630 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130184 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 10:26:00.132630 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130186 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 10:26:00.132630 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130189 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 10:26:00.132630 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130192 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 10:26:00.132630 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130195 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 10:26:00.132630 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130197 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 10:26:00.132630 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130200 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 10:26:00.132630 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130202 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 10:26:00.132630 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130205 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 10:26:00.132630 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130207 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 10:26:00.132630 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130210 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 10:26:00.132630 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130212 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 10:26:00.132630 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130215 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 10:26:00.133081 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130219 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 10:26:00.133081 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130223 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 10:26:00.133081 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130226 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 10:26:00.133081 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130229 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 10:26:00.133081 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130234 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 10:26:00.133081 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130237 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 10:26:00.133081 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130240 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 10:26:00.133081 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130243 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 10:26:00.133081 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130245 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 10:26:00.133081 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130248 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 10:26:00.133081 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130252 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 10:26:00.133081 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130254 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 10:26:00.133081 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130257 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 10:26:00.133081 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130259 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 10:26:00.133081 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130262 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 10:26:00.133081 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130265 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 10:26:00.133081 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130267 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 10:26:00.133081 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130270 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 10:26:00.133081 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130272 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 10:26:00.133081 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130275 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 10:26:00.133588 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130278 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 10:26:00.133588 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130281 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 10:26:00.133588 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130284 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 10:26:00.133588 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130286 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 10:26:00.133588 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130289 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 10:26:00.133588 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130292 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 10:26:00.133588 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130294 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 10:26:00.133588 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130297 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 10:26:00.133588 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130300 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 10:26:00.133588 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130302 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 10:26:00.133588 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130305 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 10:26:00.133588 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130308 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 10:26:00.133588 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130310 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 10:26:00.133588 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130313 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 10:26:00.133588 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130315 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 10:26:00.133588 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130318 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 10:26:00.133588 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130321 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 10:26:00.133588 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130324 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 10:26:00.133588 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130327 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 10:26:00.133588 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130331 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 10:26:00.134095 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130335 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 10:26:00.134095 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130338 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 10:26:00.134095 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130341 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 10:26:00.134095 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130343 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 10:26:00.134095 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130347 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 10:26:00.134095 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130349 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 10:26:00.134095 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130352 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 10:26:00.134095 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130355 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 10:26:00.134095 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130358 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 10:26:00.134095 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130361 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 09 10:26:00.134095 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130364 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 10:26:00.134095 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130367 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 10:26:00.134095 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130370 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 10:26:00.134095 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130372 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 10:26:00.134095 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130376 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 10:26:00.134095 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130379 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 10:26:00.134095 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130382 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 10:26:00.134095 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130384 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 10:26:00.134095 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130387 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 10:26:00.134095 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130390 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 10:26:00.134603 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130392 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 10:26:00.134603 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130395 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 10:26:00.134603 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130397 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 10:26:00.134603 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130401 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 10:26:00.134603 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130405 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 10:26:00.134603 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130408 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 10:26:00.134603 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130411 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 10:26:00.134603 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130413 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 10:26:00.134603 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130416 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 10:26:00.134603 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130418 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 10:26:00.134603 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130422 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 10:26:00.134603 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130424 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 10:26:00.134603 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130427 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 10:26:00.134603 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130429 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 10:26:00.134958 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.130434 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 10:26:00.134958 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130565 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 10:26:00.134958 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130570 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 10:26:00.134958 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130573 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 10:26:00.134958 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130576 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 10:26:00.134958 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130579 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 10:26:00.134958 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130582 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 10:26:00.134958 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130585 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 10:26:00.134958 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130588 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 10:26:00.134958 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130591 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 10:26:00.134958 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130593 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 10:26:00.134958 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130596 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 10:26:00.134958 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130599 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 10:26:00.134958 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130602 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 10:26:00.134958 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130604 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 10:26:00.135324 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130607 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 10:26:00.135324 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130610 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 10:26:00.135324 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130613 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 10:26:00.135324 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130616 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 10:26:00.135324 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130618 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 10:26:00.135324 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130621 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 10:26:00.135324 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130623 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 10:26:00.135324 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130626 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 10:26:00.135324 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130629 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 10:26:00.135324 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130631 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 10:26:00.135324 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130634 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 10:26:00.135324 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130637 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 10:26:00.135324 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130640 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 10:26:00.135324 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130642 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 10:26:00.135324 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130646 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 10:26:00.135324 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130648 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 10:26:00.135324 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130651 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 10:26:00.135324 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130654 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 10:26:00.135324 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130656 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 10:26:00.135811 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130660 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 10:26:00.135811 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130663 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 10:26:00.135811 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130666 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 10:26:00.135811 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130669 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 10:26:00.135811 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130671 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 10:26:00.135811 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130674 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 10:26:00.135811 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130677 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 10:26:00.135811 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130679 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 10:26:00.135811 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130682 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 10:26:00.135811 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130685 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 10:26:00.135811 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130687 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 10:26:00.135811 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130690 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 10:26:00.135811 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130693 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 10:26:00.135811 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130695 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 10:26:00.135811 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130698 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 10:26:00.135811 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130701 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 10:26:00.135811 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130703 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 10:26:00.135811 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130705 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 10:26:00.135811 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130708 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 10:26:00.136300 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130712 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 10:26:00.136300 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130715 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 10:26:00.136300 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130719 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 10:26:00.136300 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130721 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 10:26:00.136300 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130725 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 10:26:00.136300 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130728 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 10:26:00.136300 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130731 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 10:26:00.136300 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130733 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 10:26:00.136300 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130736 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 10:26:00.136300 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130740 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 10:26:00.136300 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130743 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 10:26:00.136300 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130745 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 10:26:00.136300 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130748 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 10:26:00.136300 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130750 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 10:26:00.136300 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130753 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 10:26:00.136300 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130756 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 10:26:00.136300 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130758 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 10:26:00.136300 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130761 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 10:26:00.136300 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130764 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 10:26:00.136836 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130766 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 10:26:00.136836 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130769 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 10:26:00.136836 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130772 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 10:26:00.136836 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130775 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 10:26:00.136836 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130777 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 10:26:00.136836 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130780 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 10:26:00.136836 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130783 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 10:26:00.136836 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130786 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 10:26:00.136836 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130788 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 10:26:00.136836 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130791 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 09 10:26:00.136836 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130794 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 10:26:00.136836 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130796 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 10:26:00.136836 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130799 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 10:26:00.136836 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130801 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 10:26:00.136836 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130804 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 10:26:00.136836 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130806 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 10:26:00.136836 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130809 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 10:26:00.136836 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130812 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 10:26:00.136836 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130814 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 10:26:00.136836 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130818 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 10:26:00.137331 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130822 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 10:26:00.137331 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130824 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 10:26:00.137331 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130827 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 10:26:00.137331 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130831 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 10:26:00.137331 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130833 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 10:26:00.137331 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130836 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 10:26:00.137331 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130839 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 10:26:00.137331 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:00.130841 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 10:26:00.137331 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.130847 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 10:26:00.137331 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.130959 2559 server.go:962] "Client rotation is on, will bootstrap in background" Sep 09 10:26:00.137331 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.134561 2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 09 10:26:00.137331 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.135396 2559 server.go:1019] "Starting client certificate rotation" Sep 09 10:26:00.137331 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.135495 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 10:26:00.137331 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.136122 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 10:26:00.137699 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.137544 2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 10:26:00.139137 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.139111 2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 10:26:00.146626 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.146606 2559 log.go:25] "Validated CRI v1 runtime API" Sep 09 10:26:00.152871 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.152856 2559 log.go:25] "Validated CRI v1 image API" Sep 09 10:26:00.153656 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.153644 2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 09 10:26:00.159137 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.159117 2559 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8ba5e7e3-0721-414b-8294-8e03eb19dbc6:/dev/nvme0n1p3 f5aade86-d8bd-4d7f-b064-03193ff42e87:/dev/nvme0n1p4] Sep 09 10:26:00.159195 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.159137 2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 09 10:26:00.164825 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.164702 2559 manager.go:216] Machine: {Timestamp:2026-09-09 10:26:00.163152748 +0000 UTC m=+0.337755063 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100497 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2897ef04ada92863ce287640a8cf76 SystemUUID:ec2897ef-04ad-a928-63ce-287640a8cf76 BootID:87f5b244-1336-488e-a7f1-28a6cf245100 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:47:8c:ef:63:9b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:47:8c:ef:63:9b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:8b:62:9d:90:51 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 09 10:26:00.164825 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.164812 2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 09 10:26:00.164941 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.164894 2559 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 09 10:26:00.167455 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.167423 2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 09 10:26:00.167614 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.167452 2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-133.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 09 10:26:00.168526 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.168515 2559 topology_manager.go:138] "Creating topology manager with none policy" Sep 09 10:26:00.168562 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.168528 2559 container_manager_linux.go:309] "Creating device plugin manager" Sep 09 10:26:00.168562 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.168536 2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 10:26:00.169813 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.169801 2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 10:26:00.170357 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.170336 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 10:26:00.171049 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.171038 2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 09 10:26:00.174125 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.174113 2559 state_mem.go:36] "Initialized new in-memory state store" Sep 09 10:26:00.174249 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.174240 2559 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 09 10:26:00.174284 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.174275 2559 kubelet.go:486] "Attempting to sync node with API server" Sep 09 10:26:00.174312 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.174295 2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 09 10:26:00.174312 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.174310 2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 09 10:26:00.174378 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.174319 2559 kubelet.go:398] "Adding apiserver pod source" Sep 09 10:26:00.174378 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.174329 2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 09 10:26:00.174710 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.174698 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 10:26:00.174757 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.174720 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 10:26:00.177320 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.177303 2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 09 10:26:00.179716 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.179693 2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 09 10:26:00.179790 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.179734 2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 09 10:26:00.179930 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.179917 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 09 10:26:00.179968 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.179937 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 09 10:26:00.179968 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.179946 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 09 10:26:00.179968 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.179955 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 09 10:26:00.179968 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.179966 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 09 10:26:00.180071 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.179975 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 09 10:26:00.180340 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.180329 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 09 10:26:00.180379 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.180342 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 09 10:26:00.180379 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.180353 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 09 10:26:00.180379 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.180361 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 09 10:26:00.180479 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.180382 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 09 10:26:00.180479 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.180394 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 09 10:26:00.181414 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.181229 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 09 10:26:00.181468 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.181416 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 09 10:26:00.185690 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.185670 2559 server.go:1268] "Started kubelet" Sep 09 10:26:00.185908 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.185670 2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-133.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 09 10:26:00.185964 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.185928 2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 09 10:26:00.185964 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.185913 2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 09 10:26:00.186046 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.185977 2559 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 09 10:26:00.186613 ip-10-0-139-133 systemd[1]: Started Kubernetes Kubelet. Sep 09 10:26:00.190759 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.190736 2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 09 10:26:00.191525 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:00.191446 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-133.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 09 10:26:00.192365 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.192345 2559 server.go:310] "Adding debug handlers to kubelet server" Sep 09 10:26:00.192986 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:00.192961 2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 09 10:26:00.195409 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:00.195387 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 09 10:26:00.196199 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:00.195347 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-133.ec2.internal.18d3a0258ff4ec80 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-133.ec2.internal,UID:ip-10-0-139-133.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-133.ec2.internal,},FirstTimestamp:2026-09-09 10:26:00.185646208 +0000 UTC m=+0.360248530,LastTimestamp:2026-09-09 10:26:00.185646208 +0000 UTC m=+0.360248530,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-133.ec2.internal,}" Sep 09 10:26:00.197296 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.197275 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 09 10:26:00.197720 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.197702 2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 09 10:26:00.198024 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:00.198004 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-133.ec2.internal\" not found" Sep 09 10:26:00.198548 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.198536 2559 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 09 10:26:00.198633 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.198625 2559 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 09 10:26:00.198790 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.198773 2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 09 10:26:00.198863 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.198794 2559 reconstruct.go:97] "Volume reconstruction finished" Sep 09 10:26:00.198863 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.198806 2559 reconciler.go:29] "Reconciler: start to sync state" Sep 09 10:26:00.199569 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.199554 2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 09 10:26:00.199569 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.199569 2559 factory.go:55] Registering systemd factory Sep 09 10:26:00.199691 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.199576 2559 factory.go:223] Registration of the systemd container factory successfully Sep 09 10:26:00.199842 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.199829 2559 factory.go:153] Registering CRI-O factory Sep 09 10:26:00.199908 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.199845 2559 factory.go:223] Registration of the crio container factory successfully Sep 09 10:26:00.199908 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.199861 2559 factory.go:103] Registering Raw factory Sep 09 10:26:00.199908 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.199888 2559 manager.go:1196] Started watching for new ooms in manager Sep 09 10:26:00.200316 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.200303 2559 manager.go:349] Starting recovery of all containers Sep 09 10:26:00.210121 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.210096 2559 manager.go:354] Recovery completed Sep 09 10:26:00.210710 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:00.210681 2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-133.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 09 10:26:00.210902 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:00.210881 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 09 10:26:00.214957 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.214942 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 10:26:00.216710 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.216689 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-133.ec2.internal" event="NodeHasSufficientMemory" Sep 09 10:26:00.216809 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.216724 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-133.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 10:26:00.216809 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.216740 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-133.ec2.internal" event="NodeHasSufficientPID" Sep 09 10:26:00.217379 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.217354 2559 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 09 10:26:00.217379 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.217361 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 10:26:00.217486 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.217366 2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 09 10:26:00.217486 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.217409 2559 state_mem.go:36] "Initialized new in-memory state store" Sep 09 10:26:00.218152 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.218136 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-133.ec2.internal" event="NodeHasSufficientMemory" Sep 09 10:26:00.218218 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.218170 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-133.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 10:26:00.218218 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.218181 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-133.ec2.internal" event="NodeHasSufficientPID" Sep 09 10:26:00.218681 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.218665 2559 policy_none.go:49] "None policy: Start" Sep 09 10:26:00.218714 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.218688 2559 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 09 10:26:00.218714 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.218703 2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 09 10:26:00.219352 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:00.219270 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-133.ec2.internal.18d3a02591cee633 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-133.ec2.internal,UID:ip-10-0-139-133.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-133.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-133.ec2.internal,},FirstTimestamp:2026-09-09 10:26:00.216708659 +0000 UTC m=+0.391310975,LastTimestamp:2026-09-09 10:26:00.216708659 +0000 UTC m=+0.391310975,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-133.ec2.internal,}" Sep 09 10:26:00.219873 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.219857 2559 policy_none.go:47] "Start" Sep 09 10:26:00.227862 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:00.227786 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-133.ec2.internal.18d3a02591cf3daa default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-133.ec2.internal,UID:ip-10-0-139-133.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-133.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-133.ec2.internal,},FirstTimestamp:2026-09-09 10:26:00.21673105 +0000 UTC m=+0.391333367,LastTimestamp:2026-09-09 10:26:00.21673105 +0000 UTC m=+0.391333367,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-133.ec2.internal,}" Sep 09 10:26:00.261380 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:00.252174 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-133.ec2.internal.18d3a02591cf74a2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-133.ec2.internal,UID:ip-10-0-139-133.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-133.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-133.ec2.internal,},FirstTimestamp:2026-09-09 10:26:00.216745122 +0000 UTC m=+0.391347438,LastTimestamp:2026-09-09 10:26:00.216745122 +0000 UTC m=+0.391347438,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-133.ec2.internal,}" Sep 09 10:26:00.261380 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.253236 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rl4bv" Sep 09 10:26:00.262187 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.262169 2559 manager.go:337] "Starting Device Plugin manager" Sep 09 10:26:00.262232 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:00.262219 2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 09 10:26:00.262269 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.262232 2559 server.go:87] "Starting device plugin registration server" Sep 09 10:26:00.262548 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.262534 2559 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 09 10:26:00.262596 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.262550 2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 09 10:26:00.262698 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.262684 2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 09 10:26:00.262782 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.262767 2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 09 10:26:00.262835 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.262784 2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 09 10:26:00.263415 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:00.263393 2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 09 10:26:00.263537 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:00.263446 2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-133.ec2.internal\" not found" Sep 09 10:26:00.266153 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.266135 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rl4bv" Sep 09 10:26:00.267217 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:00.267135 2559 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-133.ec2.internal.18d3a02591cee633\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-133.ec2.internal.18d3a02591cee633 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-133.ec2.internal,UID:ip-10-0-139-133.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-133.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-133.ec2.internal,},FirstTimestamp:2026-09-09 10:26:00.216708659 +0000 UTC m=+0.391310975,LastTimestamp:2026-09-09 10:26:00.218152023 +0000 UTC m=+0.392754340,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-133.ec2.internal,}" Sep 09 10:26:00.326478 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.326443 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 09 10:26:00.327601 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.327577 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 09 10:26:00.327722 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.327627 2559 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 09 10:26:00.327722 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.327650 2559 kubelet.go:2448] "Starting kubelet main sync loop" Sep 09 10:26:00.327722 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:00.327692 2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 09 10:26:00.331035 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.331009 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 10:26:00.363330 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.363221 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 10:26:00.364282 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.364263 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-133.ec2.internal" event="NodeHasSufficientMemory" Sep 09 10:26:00.364402 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.364300 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-133.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 10:26:00.364402 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.364314 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-133.ec2.internal" event="NodeHasSufficientPID" Sep 09 10:26:00.364402 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.364345 2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-133.ec2.internal" Sep 09 10:26:00.376671 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.376639 2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-133.ec2.internal" Sep 09 10:26:00.376671 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:00.376671 2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-133.ec2.internal\": node \"ip-10-0-139-133.ec2.internal\" not found" Sep 09 10:26:00.416616 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:00.416589 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-133.ec2.internal\" not found" Sep 09 10:26:00.428639 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.428608 2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-133.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-133.ec2.internal"] Sep 09 10:26:00.428726 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.428702 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 10:26:00.429724 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.429700 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-133.ec2.internal" event="NodeHasSufficientMemory" Sep 09 10:26:00.429853 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.429736 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-133.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 10:26:00.429853 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.429750 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-133.ec2.internal" event="NodeHasSufficientPID" Sep 09 10:26:00.431006 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.430989 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 10:26:00.431119 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.431103 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-133.ec2.internal" Sep 09 10:26:00.431167 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.431133 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 10:26:00.432898 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.432333 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-133.ec2.internal" event="NodeHasSufficientMemory" Sep 09 10:26:00.432898 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.432360 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-133.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 10:26:00.432898 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.432374 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-133.ec2.internal" event="NodeHasSufficientPID" Sep 09 10:26:00.432898 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.432645 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-133.ec2.internal" event="NodeHasSufficientMemory" Sep 09 10:26:00.432898 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.432661 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-133.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 10:26:00.432898 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.432672 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-133.ec2.internal" event="NodeHasSufficientPID" Sep 09 10:26:00.433839 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.433822 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-133.ec2.internal" Sep 09 10:26:00.433930 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.433852 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 10:26:00.434585 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.434571 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-133.ec2.internal" event="NodeHasSufficientMemory" Sep 09 10:26:00.434650 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.434597 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-133.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 10:26:00.434650 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.434609 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-133.ec2.internal" event="NodeHasSufficientPID" Sep 09 10:26:00.458456 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:00.458426 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-133.ec2.internal\" not found" node="ip-10-0-139-133.ec2.internal" Sep 09 10:26:00.462837 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:00.462818 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-133.ec2.internal\" not found" node="ip-10-0-139-133.ec2.internal" Sep 09 10:26:00.500322 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.500289 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/03c4761a17877d2c4ef55ddef5cde326-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-133.ec2.internal\" (UID: \"03c4761a17877d2c4ef55ddef5cde326\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-133.ec2.internal" Sep 09 10:26:00.500322 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.500323 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03c4761a17877d2c4ef55ddef5cde326-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-133.ec2.internal\" (UID: \"03c4761a17877d2c4ef55ddef5cde326\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-133.ec2.internal" Sep 09 10:26:00.500537 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.500348 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cecf6a195bd7af1b2519a96aa95ed769-config\") pod \"kube-apiserver-proxy-ip-10-0-139-133.ec2.internal\" (UID: \"cecf6a195bd7af1b2519a96aa95ed769\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-133.ec2.internal" Sep 09 10:26:00.517355 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:00.517324 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-133.ec2.internal\" not found" Sep 09 10:26:00.601085 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.601050 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/03c4761a17877d2c4ef55ddef5cde326-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-133.ec2.internal\" (UID: \"03c4761a17877d2c4ef55ddef5cde326\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-133.ec2.internal" Sep 09 10:26:00.601085 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.601083 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03c4761a17877d2c4ef55ddef5cde326-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-133.ec2.internal\" (UID: \"03c4761a17877d2c4ef55ddef5cde326\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-133.ec2.internal" Sep 09 10:26:00.601272 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.601102 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cecf6a195bd7af1b2519a96aa95ed769-config\") pod \"kube-apiserver-proxy-ip-10-0-139-133.ec2.internal\" (UID: \"cecf6a195bd7af1b2519a96aa95ed769\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-133.ec2.internal" Sep 09 10:26:00.601272 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.601168 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03c4761a17877d2c4ef55ddef5cde326-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-133.ec2.internal\" (UID: \"03c4761a17877d2c4ef55ddef5cde326\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-133.ec2.internal" Sep 09 10:26:00.601272 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.601169 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/03c4761a17877d2c4ef55ddef5cde326-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-133.ec2.internal\" (UID: \"03c4761a17877d2c4ef55ddef5cde326\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-133.ec2.internal" Sep 09 10:26:00.601272 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.601174 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cecf6a195bd7af1b2519a96aa95ed769-config\") pod \"kube-apiserver-proxy-ip-10-0-139-133.ec2.internal\" (UID: \"cecf6a195bd7af1b2519a96aa95ed769\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-133.ec2.internal" Sep 09 10:26:00.617844 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:00.617784 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-133.ec2.internal\" not found" Sep 09 10:26:00.718238 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:00.718200 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-133.ec2.internal\" not found" Sep 09 10:26:00.761171 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.761131 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-133.ec2.internal" Sep 09 10:26:00.766302 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:00.766285 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-133.ec2.internal" Sep 09 10:26:00.819063 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:00.819012 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-133.ec2.internal\" not found" Sep 09 10:26:00.920212 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:00.920115 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-133.ec2.internal\" not found" Sep 09 10:26:01.020301 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:01.020255 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-133.ec2.internal\" not found" Sep 09 10:26:01.116212 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.116178 2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 10:26:01.135481 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.135436 2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 09 10:26:01.135681 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.135659 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 09 10:26:01.135742 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.135683 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 09 10:26:01.174698 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.174641 2559 apiserver.go:52] "Watching apiserver" Sep 09 10:26:01.195605 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.195578 2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 10:26:01.197851 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.197833 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 09 10:26:01.198290 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.198276 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-133.ec2.internal" Sep 09 10:26:01.202492 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.202474 2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 09 10:26:01.202819 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.202799 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-k9fqd","kube-system/konnectivity-agent-l9ttv","openshift-cluster-node-tuning-operator/tuned-msgr4","openshift-dns/node-resolver-4lsnr","openshift-image-registry/node-ca-bq9w9","openshift-multus/multus-88lrx","openshift-multus/multus-additional-cni-plugins-5dpbc","openshift-network-operator/iptables-alerter-xzlmj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcg2k","openshift-multus/network-metrics-daemon-c4s48","openshift-network-diagnostics/network-check-target-vsngp","openshift-ovn-kubernetes/ovnkube-node-d22ff"] Sep 09 10:26:01.204190 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.204174 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:01.204275 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:01.204252 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9fqd" podUID="b697c056-99a2-4c91-a7c3-1f787553973e" Sep 09 10:26:01.205296 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.205281 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l9ttv" Sep 09 10:26:01.206304 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.206285 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.207304 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.207284 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4lsnr" Sep 09 10:26:01.208528 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.208477 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bq9w9" Sep 09 10:26:01.209546 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.209512 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 09 10:26:01.210637 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.209932 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9bgk6\"" Sep 09 10:26:01.210637 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.210255 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 09 10:26:01.210637 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.210494 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 09 10:26:01.210821 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.210713 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-txrf8\"" Sep 09 10:26:01.210821 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.210806 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 09 10:26:01.211058 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.211016 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 09 10:26:01.211117 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.211106 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-72tcv\"" Sep 09 10:26:01.211276 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.211258 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 09 10:26:01.211386 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.211372 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.211478 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.211373 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 09 10:26:01.211977 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.211839 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5dpbc" Sep 09 10:26:01.212710 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.212687 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rvg4l\"" Sep 09 10:26:01.213022 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.213004 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 09 10:26:01.213022 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.213015 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 09 10:26:01.214950 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.214925 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xzlmj" Sep 09 10:26:01.216336 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.216318 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 09 10:26:01.216731 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.216709 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 09 10:26:01.216816 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.216762 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rplc9\"" Sep 09 10:26:01.217550 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.217520 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 10:26:01.217638 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.217593 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-133.ec2.internal" Sep 09 10:26:01.218039 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.217994 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 09 10:26:01.218144 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.218106 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 09 10:26:01.218144 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.218119 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 09 10:26:01.218369 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.218350 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcg2k" Sep 09 10:26:01.218841 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.218764 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 09 10:26:01.218841 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.218766 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 09 10:26:01.218841 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.218826 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 09 10:26:01.219028 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.219010 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 09 10:26:01.219226 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.219209 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-t9q8x\"" Sep 09 10:26:01.219740 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.219723 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vk6hs\"" Sep 09 10:26:01.219827 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.219766 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:26:01.219879 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:01.219826 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4s48" podUID="1b8cf871-4dea-4ac5-868a-35b02f455e65" Sep 09 10:26:01.220947 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.220930 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:26:01.221031 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:01.220991 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsngp" podUID="946804f0-f29b-4f4c-8990-3cc0bf40c79e" Sep 09 10:26:01.222119 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.222101 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-133.ec2.internal"] Sep 09 10:26:01.222206 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.222192 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.222412 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.222393 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 09 10:26:01.222718 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.222693 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 09 10:26:01.222718 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.222707 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 09 10:26:01.223003 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.222986 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8g9hk\"" Sep 09 10:26:01.225304 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.225287 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 10:26:01.226196 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.226179 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bshz9\"" Sep 09 10:26:01.226526 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.226499 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 09 10:26:01.226608 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.226581 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 09 10:26:01.226744 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.226731 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 09 10:26:01.231339 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.231314 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 09 10:26:01.231411 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.231323 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 09 10:26:01.231411 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.231376 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 09 10:26:01.231602 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.231584 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-133.ec2.internal"] Sep 09 10:26:01.231708 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.231693 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 10:26:01.266739 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.266720 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hgh7d" Sep 09 10:26:01.267854 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.267806 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 10:21:00 +0000 UTC" deadline="2028-06-10 13:42:44.705692116 +0000 UTC" Sep 09 10:26:01.267914 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.267857 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15363h16m43.437839632s" Sep 09 10:26:01.276464 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.276442 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hgh7d" Sep 09 10:26:01.300842 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.300818 2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 09 10:26:01.305459 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.305436 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6aee692-b2c3-48d5-af2b-90bdeb0a29a7-system-cni-dir\") pod \"multus-additional-cni-plugins-5dpbc\" (UID: \"f6aee692-b2c3-48d5-af2b-90bdeb0a29a7\") " pod="openshift-multus/multus-additional-cni-plugins-5dpbc" Sep 09 10:26:01.305571 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.305471 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6aee692-b2c3-48d5-af2b-90bdeb0a29a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5dpbc\" (UID: \"f6aee692-b2c3-48d5-af2b-90bdeb0a29a7\") " pod="openshift-multus/multus-additional-cni-plugins-5dpbc" Sep 09 10:26:01.305571 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.305523 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-cnibin\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.305571 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.305542 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-systemd-units\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.305571 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.305555 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-host\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.305571 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.305571 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6aee692-b2c3-48d5-af2b-90bdeb0a29a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-5dpbc\" (UID: \"f6aee692-b2c3-48d5-af2b-90bdeb0a29a7\") " pod="openshift-multus/multus-additional-cni-plugins-5dpbc" Sep 09 10:26:01.305733 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.305587 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-host-var-lib-cni-bin\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.305733 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.305630 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-etc-kubernetes\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.305733 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.305660 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54scz\" (UniqueName: \"kubernetes.io/projected/946804f0-f29b-4f4c-8990-3cc0bf40c79e-kube-api-access-54scz\") pod \"network-check-target-vsngp\" (UID: \"946804f0-f29b-4f4c-8990-3cc0bf40c79e\") " pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:26:01.305733 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.305680 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-run-openvswitch\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.305733 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.305700 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-env-overrides\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.305733 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.305723 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-ovnkube-script-lib\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.305910 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.305739 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-etc-sysconfig\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.305910 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.305754 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-host-var-lib-cni-multus\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.305910 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.305776 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-run-ovn\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.305910 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.305827 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-host-cni-bin\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.305910 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.305850 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-system-cni-dir\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.305910 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.305864 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1bf313a4-2a86-47a9-8b41-934ec362c7a5-cni-binary-copy\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.305910 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.305883 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-host-run-netns\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.305910 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.305900 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6bc90a57-0901-47af-9adf-a3f4f52e7d4a-device-dir\") pod \"aws-ebs-csi-driver-node-mcg2k\" (UID: \"6bc90a57-0901-47af-9adf-a3f4f52e7d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcg2k" Sep 09 10:26:01.306129 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.305914 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-host-slash\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.306129 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.305949 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-var-lib-openvswitch\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.306129 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.305982 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxqbz\" (UniqueName: \"kubernetes.io/projected/1b8cf871-4dea-4ac5-868a-35b02f455e65-kube-api-access-bxqbz\") pod \"network-metrics-daemon-c4s48\" (UID: \"1b8cf871-4dea-4ac5-868a-35b02f455e65\") " pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:26:01.306129 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306001 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-host-cni-netd\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.306129 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306017 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psltz\" (UniqueName: \"kubernetes.io/projected/9ea95c24-1cdb-4160-a975-615f6a38bad1-kube-api-access-psltz\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.306129 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306033 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf2v6\" (UniqueName: \"kubernetes.io/projected/f6aee692-b2c3-48d5-af2b-90bdeb0a29a7-kube-api-access-bf2v6\") pod \"multus-additional-cni-plugins-5dpbc\" (UID: \"f6aee692-b2c3-48d5-af2b-90bdeb0a29a7\") " pod="openshift-multus/multus-additional-cni-plugins-5dpbc" Sep 09 10:26:01.306129 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306049 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6bc90a57-0901-47af-9adf-a3f4f52e7d4a-etc-selinux\") pod \"aws-ebs-csi-driver-node-mcg2k\" (UID: \"6bc90a57-0901-47af-9adf-a3f4f52e7d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcg2k" Sep 09 10:26:01.306129 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306064 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-log-socket\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.306129 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306079 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tzmm\" (UniqueName: \"kubernetes.io/projected/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-kube-api-access-7tzmm\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.306129 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306093 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-lib-modules\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.306129 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306115 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ea95c24-1cdb-4160-a975-615f6a38bad1-etc-tuned\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.306129 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306133 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-multus-cni-dir\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.306522 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306151 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-etc-openvswitch\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.306522 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306174 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-ovn-node-metrics-cert\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.306522 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306193 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b697c056-99a2-4c91-a7c3-1f787553973e-original-pull-secret\") pod \"global-pull-secret-syncer-k9fqd\" (UID: \"b697c056-99a2-4c91-a7c3-1f787553973e\") " pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:01.306522 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306216 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-etc-kubernetes\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.306522 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306233 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-etc-sysctl-conf\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.306522 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306255 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx5h9\" (UniqueName: \"kubernetes.io/projected/13c59e77-04fc-4549-8402-df18b180161f-kube-api-access-cx5h9\") pod \"node-resolver-4lsnr\" (UID: \"13c59e77-04fc-4549-8402-df18b180161f\") " pod="openshift-dns/node-resolver-4lsnr" Sep 09 10:26:01.306522 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306276 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f6aee692-b2c3-48d5-af2b-90bdeb0a29a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5dpbc\" (UID: \"f6aee692-b2c3-48d5-af2b-90bdeb0a29a7\") " pod="openshift-multus/multus-additional-cni-plugins-5dpbc" Sep 09 10:26:01.306522 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306301 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-host-var-lib-kubelet\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.306522 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306321 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.306522 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306343 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ea95c24-1cdb-4160-a975-615f6a38bad1-tmp\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.306522 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306364 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6bc90a57-0901-47af-9adf-a3f4f52e7d4a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mcg2k\" (UID: \"6bc90a57-0901-47af-9adf-a3f4f52e7d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcg2k" Sep 09 10:26:01.306522 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306380 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6bc90a57-0901-47af-9adf-a3f4f52e7d4a-socket-dir\") pod \"aws-ebs-csi-driver-node-mcg2k\" (UID: \"6bc90a57-0901-47af-9adf-a3f4f52e7d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcg2k" Sep 09 10:26:01.306522 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306394 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1bf313a4-2a86-47a9-8b41-934ec362c7a5-multus-daemon-config\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.306522 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306413 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-ovnkube-config\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.306522 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306436 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-etc-modprobe-d\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.306522 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306455 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-etc-systemd\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.307202 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306472 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e342f56b-7a5f-4175-bc85-285df46a3475-host-slash\") pod \"iptables-alerter-xzlmj\" (UID: \"e342f56b-7a5f-4175-bc85-285df46a3475\") " pod="openshift-network-operator/iptables-alerter-xzlmj" Sep 09 10:26:01.307202 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306495 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3ceff104-498a-49b9-a2b1-0d9de58ebca3-konnectivity-ca\") pod \"konnectivity-agent-l9ttv\" (UID: \"3ceff104-498a-49b9-a2b1-0d9de58ebca3\") " pod="kube-system/konnectivity-agent-l9ttv" Sep 09 10:26:01.307202 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306530 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b697c056-99a2-4c91-a7c3-1f787553973e-kubelet-config\") pod \"global-pull-secret-syncer-k9fqd\" (UID: \"b697c056-99a2-4c91-a7c3-1f787553973e\") " pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:01.307202 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306546 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/784872fd-624e-4323-862a-f1a040222214-serviceca\") pod \"node-ca-bq9w9\" (UID: \"784872fd-624e-4323-862a-f1a040222214\") " pod="openshift-image-registry/node-ca-bq9w9" Sep 09 10:26:01.307202 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306560 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6aee692-b2c3-48d5-af2b-90bdeb0a29a7-cnibin\") pod \"multus-additional-cni-plugins-5dpbc\" (UID: \"f6aee692-b2c3-48d5-af2b-90bdeb0a29a7\") " pod="openshift-multus/multus-additional-cni-plugins-5dpbc" Sep 09 10:26:01.307202 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306577 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f6aee692-b2c3-48d5-af2b-90bdeb0a29a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5dpbc\" (UID: \"f6aee692-b2c3-48d5-af2b-90bdeb0a29a7\") " pod="openshift-multus/multus-additional-cni-plugins-5dpbc" Sep 09 10:26:01.307202 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306598 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e342f56b-7a5f-4175-bc85-285df46a3475-iptables-alerter-script\") pod \"iptables-alerter-xzlmj\" (UID: \"e342f56b-7a5f-4175-bc85-285df46a3475\") " pod="openshift-network-operator/iptables-alerter-xzlmj" Sep 09 10:26:01.307202 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306640 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6bc90a57-0901-47af-9adf-a3f4f52e7d4a-registration-dir\") pod \"aws-ebs-csi-driver-node-mcg2k\" (UID: \"6bc90a57-0901-47af-9adf-a3f4f52e7d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcg2k" Sep 09 10:26:01.307202 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306657 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6bc90a57-0901-47af-9adf-a3f4f52e7d4a-sys-fs\") pod \"aws-ebs-csi-driver-node-mcg2k\" (UID: \"6bc90a57-0901-47af-9adf-a3f4f52e7d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcg2k" Sep 09 10:26:01.307202 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306671 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9brf7\" (UniqueName: \"kubernetes.io/projected/6bc90a57-0901-47af-9adf-a3f4f52e7d4a-kube-api-access-9brf7\") pod \"aws-ebs-csi-driver-node-mcg2k\" (UID: \"6bc90a57-0901-47af-9adf-a3f4f52e7d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcg2k" Sep 09 10:26:01.307202 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306685 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-etc-sysctl-d\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.307202 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306699 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/784872fd-624e-4323-862a-f1a040222214-host\") pod \"node-ca-bq9w9\" (UID: \"784872fd-624e-4323-862a-f1a040222214\") " pod="openshift-image-registry/node-ca-bq9w9" Sep 09 10:26:01.307202 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306713 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxxt7\" (UniqueName: \"kubernetes.io/projected/784872fd-624e-4323-862a-f1a040222214-kube-api-access-pxxt7\") pod \"node-ca-bq9w9\" (UID: \"784872fd-624e-4323-862a-f1a040222214\") " pod="openshift-image-registry/node-ca-bq9w9" Sep 09 10:26:01.307202 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306726 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-host-run-k8s-cni-cncf-io\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.307202 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306741 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-multus-conf-dir\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.307202 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306755 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs\") pod \"network-metrics-daemon-c4s48\" (UID: \"1b8cf871-4dea-4ac5-868a-35b02f455e65\") " pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:26:01.307734 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306770 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-host-kubelet\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.307734 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306783 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-node-log\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.307734 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306797 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-host-run-ovn-kubernetes\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.307734 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306811 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-sys\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.307734 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306843 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js8gv\" (UniqueName: \"kubernetes.io/projected/e342f56b-7a5f-4175-bc85-285df46a3475-kube-api-access-js8gv\") pod \"iptables-alerter-xzlmj\" (UID: \"e342f56b-7a5f-4175-bc85-285df46a3475\") " pod="openshift-network-operator/iptables-alerter-xzlmj" Sep 09 10:26:01.307734 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306857 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3ceff104-498a-49b9-a2b1-0d9de58ebca3-agent-certs\") pod \"konnectivity-agent-l9ttv\" (UID: \"3ceff104-498a-49b9-a2b1-0d9de58ebca3\") " pod="kube-system/konnectivity-agent-l9ttv" Sep 09 10:26:01.307734 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306894 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-multus-socket-dir-parent\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.307734 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306913 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-host-run-multus-certs\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.307734 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306930 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vl9d\" (UniqueName: \"kubernetes.io/projected/1bf313a4-2a86-47a9-8b41-934ec362c7a5-kube-api-access-6vl9d\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.307734 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306951 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b697c056-99a2-4c91-a7c3-1f787553973e-dbus\") pod \"global-pull-secret-syncer-k9fqd\" (UID: \"b697c056-99a2-4c91-a7c3-1f787553973e\") " pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:01.307734 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306966 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/13c59e77-04fc-4549-8402-df18b180161f-hosts-file\") pod \"node-resolver-4lsnr\" (UID: \"13c59e77-04fc-4549-8402-df18b180161f\") " pod="openshift-dns/node-resolver-4lsnr" Sep 09 10:26:01.307734 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306979 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-hostroot\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.307734 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.306992 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-host-run-netns\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.307734 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.307033 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-run-systemd\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.307734 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.307067 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-run\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.307734 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.307094 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-var-lib-kubelet\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.307734 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.307132 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c59e77-04fc-4549-8402-df18b180161f-tmp-dir\") pod \"node-resolver-4lsnr\" (UID: \"13c59e77-04fc-4549-8402-df18b180161f\") " pod="openshift-dns/node-resolver-4lsnr" Sep 09 10:26:01.407946 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.407913 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-run-openvswitch\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.407946 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.407946 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-env-overrides\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.408179 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.407963 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-ovnkube-script-lib\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.408179 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.407995 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-etc-sysconfig\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.408179 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408015 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-host-var-lib-cni-multus\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.408179 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408040 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-run-ovn\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.408179 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408037 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-run-openvswitch\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.408179 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408078 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-run-ovn\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.408179 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408084 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-etc-sysconfig\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.408179 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408098 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-host-var-lib-cni-multus\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.408179 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408128 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-host-cni-bin\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.408179 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408173 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-system-cni-dir\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.408685 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408197 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1bf313a4-2a86-47a9-8b41-934ec362c7a5-cni-binary-copy\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.408685 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408224 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-host-run-netns\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.408685 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408278 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-host-run-netns\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.408685 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408270 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-host-cni-bin\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.408685 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408320 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6bc90a57-0901-47af-9adf-a3f4f52e7d4a-device-dir\") pod \"aws-ebs-csi-driver-node-mcg2k\" (UID: \"6bc90a57-0901-47af-9adf-a3f4f52e7d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcg2k" Sep 09 10:26:01.408685 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408303 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-system-cni-dir\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.408685 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408347 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-host-slash\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.408685 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408372 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-var-lib-openvswitch\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.408685 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408385 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6bc90a57-0901-47af-9adf-a3f4f52e7d4a-device-dir\") pod \"aws-ebs-csi-driver-node-mcg2k\" (UID: \"6bc90a57-0901-47af-9adf-a3f4f52e7d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcg2k" Sep 09 10:26:01.408685 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408418 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-var-lib-openvswitch\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.408685 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408415 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-host-slash\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.408685 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408439 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxqbz\" (UniqueName: \"kubernetes.io/projected/1b8cf871-4dea-4ac5-868a-35b02f455e65-kube-api-access-bxqbz\") pod \"network-metrics-daemon-c4s48\" (UID: \"1b8cf871-4dea-4ac5-868a-35b02f455e65\") " pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:26:01.408685 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408470 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-host-cni-netd\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.408685 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408491 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-env-overrides\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.408685 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408495 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psltz\" (UniqueName: \"kubernetes.io/projected/9ea95c24-1cdb-4160-a975-615f6a38bad1-kube-api-access-psltz\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.408685 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408534 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-host-cni-netd\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.408685 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408554 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf2v6\" (UniqueName: \"kubernetes.io/projected/f6aee692-b2c3-48d5-af2b-90bdeb0a29a7-kube-api-access-bf2v6\") pod \"multus-additional-cni-plugins-5dpbc\" (UID: \"f6aee692-b2c3-48d5-af2b-90bdeb0a29a7\") " pod="openshift-multus/multus-additional-cni-plugins-5dpbc" Sep 09 10:26:01.409468 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408580 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6bc90a57-0901-47af-9adf-a3f4f52e7d4a-etc-selinux\") pod \"aws-ebs-csi-driver-node-mcg2k\" (UID: \"6bc90a57-0901-47af-9adf-a3f4f52e7d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcg2k" Sep 09 10:26:01.409468 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408602 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-log-socket\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.409468 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408625 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tzmm\" (UniqueName: \"kubernetes.io/projected/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-kube-api-access-7tzmm\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.409468 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408649 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-lib-modules\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.409468 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408687 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ea95c24-1cdb-4160-a975-615f6a38bad1-etc-tuned\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.409468 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408691 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-ovnkube-script-lib\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.409468 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408711 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-multus-cni-dir\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.409468 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408687 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6bc90a57-0901-47af-9adf-a3f4f52e7d4a-etc-selinux\") pod \"aws-ebs-csi-driver-node-mcg2k\" (UID: \"6bc90a57-0901-47af-9adf-a3f4f52e7d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcg2k" Sep 09 10:26:01.409468 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408761 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-etc-openvswitch\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.409468 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408799 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-ovn-node-metrics-cert\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.409468 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408776 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-multus-cni-dir\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.409468 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408807 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1bf313a4-2a86-47a9-8b41-934ec362c7a5-cni-binary-copy\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.409468 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408827 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b697c056-99a2-4c91-a7c3-1f787553973e-original-pull-secret\") pod \"global-pull-secret-syncer-k9fqd\" (UID: \"b697c056-99a2-4c91-a7c3-1f787553973e\") " pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:01.409468 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408846 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-etc-openvswitch\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.409468 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408857 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-etc-kubernetes\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.409468 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408888 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-etc-sysctl-conf\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.409468 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408900 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-lib-modules\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.410304 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408916 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx5h9\" (UniqueName: \"kubernetes.io/projected/13c59e77-04fc-4549-8402-df18b180161f-kube-api-access-cx5h9\") pod \"node-resolver-4lsnr\" (UID: \"13c59e77-04fc-4549-8402-df18b180161f\") " pod="openshift-dns/node-resolver-4lsnr" Sep 09 10:26:01.410304 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408943 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f6aee692-b2c3-48d5-af2b-90bdeb0a29a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5dpbc\" (UID: \"f6aee692-b2c3-48d5-af2b-90bdeb0a29a7\") " pod="openshift-multus/multus-additional-cni-plugins-5dpbc" Sep 09 10:26:01.410304 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408918 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-etc-kubernetes\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.410304 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408966 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-host-var-lib-kubelet\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.410304 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.408989 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.410304 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409010 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ea95c24-1cdb-4160-a975-615f6a38bad1-tmp\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.410304 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409030 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-etc-sysctl-conf\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.410304 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409036 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6bc90a57-0901-47af-9adf-a3f4f52e7d4a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mcg2k\" (UID: \"6bc90a57-0901-47af-9adf-a3f4f52e7d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcg2k" Sep 09 10:26:01.410304 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:01.408945 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 10:26:01.410304 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409056 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-host-var-lib-kubelet\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.410304 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409061 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6bc90a57-0901-47af-9adf-a3f4f52e7d4a-socket-dir\") pod \"aws-ebs-csi-driver-node-mcg2k\" (UID: \"6bc90a57-0901-47af-9adf-a3f4f52e7d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcg2k" Sep 09 10:26:01.410304 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409082 2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 09 10:26:01.410304 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:01.409130 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b697c056-99a2-4c91-a7c3-1f787553973e-original-pull-secret podName:b697c056-99a2-4c91-a7c3-1f787553973e nodeName:}" failed. No retries permitted until 2026-09-09 10:26:01.909089604 +0000 UTC m=+2.083691928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b697c056-99a2-4c91-a7c3-1f787553973e-original-pull-secret") pod "global-pull-secret-syncer-k9fqd" (UID: "b697c056-99a2-4c91-a7c3-1f787553973e") : object "kube-system"/"original-pull-secret" not registered Sep 09 10:26:01.410304 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409150 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1bf313a4-2a86-47a9-8b41-934ec362c7a5-multus-daemon-config\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.410304 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409160 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6bc90a57-0901-47af-9adf-a3f4f52e7d4a-socket-dir\") pod \"aws-ebs-csi-driver-node-mcg2k\" (UID: \"6bc90a57-0901-47af-9adf-a3f4f52e7d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcg2k" Sep 09 10:26:01.410304 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409163 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.410304 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409178 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-ovnkube-config\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.411052 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409168 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6bc90a57-0901-47af-9adf-a3f4f52e7d4a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mcg2k\" (UID: \"6bc90a57-0901-47af-9adf-a3f4f52e7d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcg2k" Sep 09 10:26:01.411052 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409376 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-etc-modprobe-d\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.411052 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409417 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-etc-systemd\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.411052 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409441 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e342f56b-7a5f-4175-bc85-285df46a3475-host-slash\") pod \"iptables-alerter-xzlmj\" (UID: \"e342f56b-7a5f-4175-bc85-285df46a3475\") " pod="openshift-network-operator/iptables-alerter-xzlmj" Sep 09 10:26:01.411052 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409467 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3ceff104-498a-49b9-a2b1-0d9de58ebca3-konnectivity-ca\") pod \"konnectivity-agent-l9ttv\" (UID: \"3ceff104-498a-49b9-a2b1-0d9de58ebca3\") " pod="kube-system/konnectivity-agent-l9ttv" Sep 09 10:26:01.411052 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409491 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b697c056-99a2-4c91-a7c3-1f787553973e-kubelet-config\") pod \"global-pull-secret-syncer-k9fqd\" (UID: \"b697c056-99a2-4c91-a7c3-1f787553973e\") " pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:01.411052 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409499 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f6aee692-b2c3-48d5-af2b-90bdeb0a29a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5dpbc\" (UID: \"f6aee692-b2c3-48d5-af2b-90bdeb0a29a7\") " pod="openshift-multus/multus-additional-cni-plugins-5dpbc" Sep 09 10:26:01.411052 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409512 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/784872fd-624e-4323-862a-f1a040222214-serviceca\") pod \"node-ca-bq9w9\" (UID: \"784872fd-624e-4323-862a-f1a040222214\") " pod="openshift-image-registry/node-ca-bq9w9" Sep 09 10:26:01.411052 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409550 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6aee692-b2c3-48d5-af2b-90bdeb0a29a7-cnibin\") pod \"multus-additional-cni-plugins-5dpbc\" (UID: \"f6aee692-b2c3-48d5-af2b-90bdeb0a29a7\") " pod="openshift-multus/multus-additional-cni-plugins-5dpbc" Sep 09 10:26:01.411052 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409572 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f6aee692-b2c3-48d5-af2b-90bdeb0a29a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5dpbc\" (UID: \"f6aee692-b2c3-48d5-af2b-90bdeb0a29a7\") " pod="openshift-multus/multus-additional-cni-plugins-5dpbc" Sep 09 10:26:01.411052 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409577 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e342f56b-7a5f-4175-bc85-285df46a3475-host-slash\") pod \"iptables-alerter-xzlmj\" (UID: \"e342f56b-7a5f-4175-bc85-285df46a3475\") " pod="openshift-network-operator/iptables-alerter-xzlmj" Sep 09 10:26:01.411052 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409595 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e342f56b-7a5f-4175-bc85-285df46a3475-iptables-alerter-script\") pod \"iptables-alerter-xzlmj\" (UID: \"e342f56b-7a5f-4175-bc85-285df46a3475\") " pod="openshift-network-operator/iptables-alerter-xzlmj" Sep 09 10:26:01.411052 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409616 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6bc90a57-0901-47af-9adf-a3f4f52e7d4a-registration-dir\") pod \"aws-ebs-csi-driver-node-mcg2k\" (UID: \"6bc90a57-0901-47af-9adf-a3f4f52e7d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcg2k" Sep 09 10:26:01.411052 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409635 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6bc90a57-0901-47af-9adf-a3f4f52e7d4a-sys-fs\") pod \"aws-ebs-csi-driver-node-mcg2k\" (UID: \"6bc90a57-0901-47af-9adf-a3f4f52e7d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcg2k" Sep 09 10:26:01.411052 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409653 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-etc-modprobe-d\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.411052 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409656 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9brf7\" (UniqueName: \"kubernetes.io/projected/6bc90a57-0901-47af-9adf-a3f4f52e7d4a-kube-api-access-9brf7\") pod \"aws-ebs-csi-driver-node-mcg2k\" (UID: \"6bc90a57-0901-47af-9adf-a3f4f52e7d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcg2k" Sep 09 10:26:01.411052 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409680 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-etc-sysctl-d\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.411965 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409697 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-etc-systemd\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.411965 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409701 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/784872fd-624e-4323-862a-f1a040222214-host\") pod \"node-ca-bq9w9\" (UID: \"784872fd-624e-4323-862a-f1a040222214\") " pod="openshift-image-registry/node-ca-bq9w9" Sep 09 10:26:01.411965 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409727 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxxt7\" (UniqueName: \"kubernetes.io/projected/784872fd-624e-4323-862a-f1a040222214-kube-api-access-pxxt7\") pod \"node-ca-bq9w9\" (UID: \"784872fd-624e-4323-862a-f1a040222214\") " pod="openshift-image-registry/node-ca-bq9w9" Sep 09 10:26:01.411965 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409726 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-ovnkube-config\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.411965 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409739 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b697c056-99a2-4c91-a7c3-1f787553973e-kubelet-config\") pod \"global-pull-secret-syncer-k9fqd\" (UID: \"b697c056-99a2-4c91-a7c3-1f787553973e\") " pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:01.411965 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409747 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-host-run-k8s-cni-cncf-io\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.411965 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409773 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-multus-conf-dir\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.411965 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409820 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-multus-conf-dir\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.411965 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409904 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-etc-sysctl-d\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.411965 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.409662 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1bf313a4-2a86-47a9-8b41-934ec362c7a5-multus-daemon-config\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.411965 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.410012 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/784872fd-624e-4323-862a-f1a040222214-host\") pod \"node-ca-bq9w9\" (UID: \"784872fd-624e-4323-862a-f1a040222214\") " pod="openshift-image-registry/node-ca-bq9w9" Sep 09 10:26:01.411965 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.410052 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-host-run-k8s-cni-cncf-io\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.411965 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.410095 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f6aee692-b2c3-48d5-af2b-90bdeb0a29a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5dpbc\" (UID: \"f6aee692-b2c3-48d5-af2b-90bdeb0a29a7\") " pod="openshift-multus/multus-additional-cni-plugins-5dpbc" Sep 09 10:26:01.411965 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.410147 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6bc90a57-0901-47af-9adf-a3f4f52e7d4a-sys-fs\") pod \"aws-ebs-csi-driver-node-mcg2k\" (UID: \"6bc90a57-0901-47af-9adf-a3f4f52e7d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcg2k" Sep 09 10:26:01.411965 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.410191 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3ceff104-498a-49b9-a2b1-0d9de58ebca3-konnectivity-ca\") pod \"konnectivity-agent-l9ttv\" (UID: \"3ceff104-498a-49b9-a2b1-0d9de58ebca3\") " pod="kube-system/konnectivity-agent-l9ttv" Sep 09 10:26:01.411965 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.410209 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6bc90a57-0901-47af-9adf-a3f4f52e7d4a-registration-dir\") pod \"aws-ebs-csi-driver-node-mcg2k\" (UID: \"6bc90a57-0901-47af-9adf-a3f4f52e7d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcg2k" Sep 09 10:26:01.411965 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.410239 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6aee692-b2c3-48d5-af2b-90bdeb0a29a7-cnibin\") pod \"multus-additional-cni-plugins-5dpbc\" (UID: \"f6aee692-b2c3-48d5-af2b-90bdeb0a29a7\") " pod="openshift-multus/multus-additional-cni-plugins-5dpbc" Sep 09 10:26:01.411965 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.410271 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs\") pod \"network-metrics-daemon-c4s48\" (UID: \"1b8cf871-4dea-4ac5-868a-35b02f455e65\") " pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:26:01.412885 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.410307 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-host-kubelet\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.412885 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.410347 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-node-log\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.412885 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.410381 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-host-run-ovn-kubernetes\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.412885 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.410400 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e342f56b-7a5f-4175-bc85-285df46a3475-iptables-alerter-script\") pod \"iptables-alerter-xzlmj\" (UID: \"e342f56b-7a5f-4175-bc85-285df46a3475\") " pod="openshift-network-operator/iptables-alerter-xzlmj" Sep 09 10:26:01.412885 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.410403 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-sys\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.412885 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.410444 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-sys\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.412885 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:01.410455 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 10:26:01.412885 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.410478 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js8gv\" (UniqueName: \"kubernetes.io/projected/e342f56b-7a5f-4175-bc85-285df46a3475-kube-api-access-js8gv\") pod \"iptables-alerter-xzlmj\" (UID: \"e342f56b-7a5f-4175-bc85-285df46a3475\") " pod="openshift-network-operator/iptables-alerter-xzlmj" Sep 09 10:26:01.412885 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:01.410508 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs podName:1b8cf871-4dea-4ac5-868a-35b02f455e65 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:01.910490902 +0000 UTC m=+2.085093210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs") pod "network-metrics-daemon-c4s48" (UID: "1b8cf871-4dea-4ac5-868a-35b02f455e65") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 10:26:01.412885 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.410529 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3ceff104-498a-49b9-a2b1-0d9de58ebca3-agent-certs\") pod \"konnectivity-agent-l9ttv\" (UID: \"3ceff104-498a-49b9-a2b1-0d9de58ebca3\") " pod="kube-system/konnectivity-agent-l9ttv" Sep 09 10:26:01.412885 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.410567 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-log-socket\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.412885 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.410689 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-host-kubelet\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.412885 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.410723 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-host-run-ovn-kubernetes\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.412885 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.410761 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-node-log\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.412885 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.410917 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-multus-socket-dir-parent\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.412885 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.410729 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-multus-socket-dir-parent\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.412885 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.411095 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-host-run-multus-certs\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.413439 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.411123 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vl9d\" (UniqueName: \"kubernetes.io/projected/1bf313a4-2a86-47a9-8b41-934ec362c7a5-kube-api-access-6vl9d\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.413439 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.411150 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b697c056-99a2-4c91-a7c3-1f787553973e-dbus\") pod \"global-pull-secret-syncer-k9fqd\" (UID: \"b697c056-99a2-4c91-a7c3-1f787553973e\") " pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:01.413439 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.411178 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/13c59e77-04fc-4549-8402-df18b180161f-hosts-file\") pod \"node-resolver-4lsnr\" (UID: \"13c59e77-04fc-4549-8402-df18b180161f\") " pod="openshift-dns/node-resolver-4lsnr" Sep 09 10:26:01.413439 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.411203 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-hostroot\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.413439 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.411205 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/784872fd-624e-4323-862a-f1a040222214-serviceca\") pod \"node-ca-bq9w9\" (UID: \"784872fd-624e-4323-862a-f1a040222214\") " pod="openshift-image-registry/node-ca-bq9w9" Sep 09 10:26:01.413439 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.411226 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-host-run-netns\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.413439 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.411288 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-host-run-netns\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.413439 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.411294 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b697c056-99a2-4c91-a7c3-1f787553973e-dbus\") pod \"global-pull-secret-syncer-k9fqd\" (UID: \"b697c056-99a2-4c91-a7c3-1f787553973e\") " pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:01.413439 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.411328 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-run-systemd\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.413439 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.411342 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-host-run-multus-certs\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.413439 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.411352 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-run\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.413439 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.411368 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-hostroot\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.413439 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.411376 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-var-lib-kubelet\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.413439 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.411425 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-var-lib-kubelet\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.413439 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.411430 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/13c59e77-04fc-4549-8402-df18b180161f-hosts-file\") pod \"node-resolver-4lsnr\" (UID: \"13c59e77-04fc-4549-8402-df18b180161f\") " pod="openshift-dns/node-resolver-4lsnr" Sep 09 10:26:01.413439 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.411436 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c59e77-04fc-4549-8402-df18b180161f-tmp-dir\") pod \"node-resolver-4lsnr\" (UID: \"13c59e77-04fc-4549-8402-df18b180161f\") " pod="openshift-dns/node-resolver-4lsnr" Sep 09 10:26:01.413439 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.411470 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6aee692-b2c3-48d5-af2b-90bdeb0a29a7-system-cni-dir\") pod \"multus-additional-cni-plugins-5dpbc\" (UID: \"f6aee692-b2c3-48d5-af2b-90bdeb0a29a7\") " pod="openshift-multus/multus-additional-cni-plugins-5dpbc" Sep 09 10:26:01.413439 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.411471 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-run-systemd\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.414136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.411526 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6aee692-b2c3-48d5-af2b-90bdeb0a29a7-system-cni-dir\") pod \"multus-additional-cni-plugins-5dpbc\" (UID: \"f6aee692-b2c3-48d5-af2b-90bdeb0a29a7\") " pod="openshift-multus/multus-additional-cni-plugins-5dpbc" Sep 09 10:26:01.414136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.411588 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-run\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.414136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.411685 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6aee692-b2c3-48d5-af2b-90bdeb0a29a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5dpbc\" (UID: \"f6aee692-b2c3-48d5-af2b-90bdeb0a29a7\") " pod="openshift-multus/multus-additional-cni-plugins-5dpbc" Sep 09 10:26:01.414136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.411711 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-cnibin\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.414136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.411736 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-systemd-units\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.414136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.411757 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-host\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.414136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.411779 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6aee692-b2c3-48d5-af2b-90bdeb0a29a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-5dpbc\" (UID: \"f6aee692-b2c3-48d5-af2b-90bdeb0a29a7\") " pod="openshift-multus/multus-additional-cni-plugins-5dpbc" Sep 09 10:26:01.414136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.411798 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c59e77-04fc-4549-8402-df18b180161f-tmp-dir\") pod \"node-resolver-4lsnr\" (UID: \"13c59e77-04fc-4549-8402-df18b180161f\") " pod="openshift-dns/node-resolver-4lsnr" Sep 09 10:26:01.414136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.411906 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-host-var-lib-cni-bin\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.414136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.411801 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-host-var-lib-cni-bin\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.414136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.411953 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6aee692-b2c3-48d5-af2b-90bdeb0a29a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5dpbc\" (UID: \"f6aee692-b2c3-48d5-af2b-90bdeb0a29a7\") " pod="openshift-multus/multus-additional-cni-plugins-5dpbc" Sep 09 10:26:01.414136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.411966 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-etc-kubernetes\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.414136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.412011 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-cnibin\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.414136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.412012 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54scz\" (UniqueName: \"kubernetes.io/projected/946804f0-f29b-4f4c-8990-3cc0bf40c79e-kube-api-access-54scz\") pod \"network-check-target-vsngp\" (UID: \"946804f0-f29b-4f4c-8990-3cc0bf40c79e\") " pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:26:01.414136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.412090 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-systemd-units\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.414136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.412141 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ea95c24-1cdb-4160-a975-615f6a38bad1-host\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.414136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.412204 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bf313a4-2a86-47a9-8b41-934ec362c7a5-etc-kubernetes\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.414136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.412723 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6aee692-b2c3-48d5-af2b-90bdeb0a29a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-5dpbc\" (UID: \"f6aee692-b2c3-48d5-af2b-90bdeb0a29a7\") " pod="openshift-multus/multus-additional-cni-plugins-5dpbc" Sep 09 10:26:01.414656 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.412943 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ea95c24-1cdb-4160-a975-615f6a38bad1-etc-tuned\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.414656 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.413087 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ea95c24-1cdb-4160-a975-615f6a38bad1-tmp\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.414656 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.413177 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-ovn-node-metrics-cert\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.414656 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.413275 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3ceff104-498a-49b9-a2b1-0d9de58ebca3-agent-certs\") pod \"konnectivity-agent-l9ttv\" (UID: \"3ceff104-498a-49b9-a2b1-0d9de58ebca3\") " pod="kube-system/konnectivity-agent-l9ttv" Sep 09 10:26:01.423727 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.423695 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 10:26:01.425322 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.425265 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf2v6\" (UniqueName: \"kubernetes.io/projected/f6aee692-b2c3-48d5-af2b-90bdeb0a29a7-kube-api-access-bf2v6\") pod \"multus-additional-cni-plugins-5dpbc\" (UID: \"f6aee692-b2c3-48d5-af2b-90bdeb0a29a7\") " pod="openshift-multus/multus-additional-cni-plugins-5dpbc" Sep 09 10:26:01.429854 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:01.429828 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 10:26:01.429854 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:01.429854 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 10:26:01.430045 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:01.429866 2559 projected.go:196] Error preparing data for projected volume kube-api-access-54scz for pod openshift-network-diagnostics/network-check-target-vsngp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 10:26:01.430045 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:01.429931 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/946804f0-f29b-4f4c-8990-3cc0bf40c79e-kube-api-access-54scz podName:946804f0-f29b-4f4c-8990-3cc0bf40c79e nodeName:}" failed. No retries permitted until 2026-09-09 10:26:01.929909959 +0000 UTC m=+2.104512279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-54scz" (UniqueName: "kubernetes.io/projected/946804f0-f29b-4f4c-8990-3cc0bf40c79e-kube-api-access-54scz") pod "network-check-target-vsngp" (UID: "946804f0-f29b-4f4c-8990-3cc0bf40c79e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 10:26:01.438014 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.437989 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxxt7\" (UniqueName: \"kubernetes.io/projected/784872fd-624e-4323-862a-f1a040222214-kube-api-access-pxxt7\") pod \"node-ca-bq9w9\" (UID: \"784872fd-624e-4323-862a-f1a040222214\") " pod="openshift-image-registry/node-ca-bq9w9" Sep 09 10:26:01.438270 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.438241 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-js8gv\" (UniqueName: \"kubernetes.io/projected/e342f56b-7a5f-4175-bc85-285df46a3475-kube-api-access-js8gv\") pod \"iptables-alerter-xzlmj\" (UID: \"e342f56b-7a5f-4175-bc85-285df46a3475\") " pod="openshift-network-operator/iptables-alerter-xzlmj" Sep 09 10:26:01.438397 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.438241 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tzmm\" (UniqueName: \"kubernetes.io/projected/0a56b285-3f3c-4f6d-a939-8a95ae00e8da-kube-api-access-7tzmm\") pod \"ovnkube-node-d22ff\" (UID: \"0a56b285-3f3c-4f6d-a939-8a95ae00e8da\") " pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.440037 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.440009 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psltz\" (UniqueName: \"kubernetes.io/projected/9ea95c24-1cdb-4160-a975-615f6a38bad1-kube-api-access-psltz\") pod \"tuned-msgr4\" (UID: \"9ea95c24-1cdb-4160-a975-615f6a38bad1\") " pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.441806 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.441772 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9brf7\" (UniqueName: \"kubernetes.io/projected/6bc90a57-0901-47af-9adf-a3f4f52e7d4a-kube-api-access-9brf7\") pod \"aws-ebs-csi-driver-node-mcg2k\" (UID: \"6bc90a57-0901-47af-9adf-a3f4f52e7d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcg2k" Sep 09 10:26:01.442837 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.442815 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx5h9\" (UniqueName: \"kubernetes.io/projected/13c59e77-04fc-4549-8402-df18b180161f-kube-api-access-cx5h9\") pod \"node-resolver-4lsnr\" (UID: \"13c59e77-04fc-4549-8402-df18b180161f\") " pod="openshift-dns/node-resolver-4lsnr" Sep 09 10:26:01.442917 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.442846 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxqbz\" (UniqueName: \"kubernetes.io/projected/1b8cf871-4dea-4ac5-868a-35b02f455e65-kube-api-access-bxqbz\") pod \"network-metrics-daemon-c4s48\" (UID: \"1b8cf871-4dea-4ac5-868a-35b02f455e65\") " pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:26:01.443280 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.443263 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vl9d\" (UniqueName: \"kubernetes.io/projected/1bf313a4-2a86-47a9-8b41-934ec362c7a5-kube-api-access-6vl9d\") pod \"multus-88lrx\" (UID: \"1bf313a4-2a86-47a9-8b41-934ec362c7a5\") " pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.521860 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.521823 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l9ttv" Sep 09 10:26:01.546876 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.546841 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-msgr4" Sep 09 10:26:01.567337 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.567298 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4lsnr" Sep 09 10:26:01.573330 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.573302 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bq9w9" Sep 09 10:26:01.586680 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.586648 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-88lrx" Sep 09 10:26:01.593248 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:01.593199 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea95c24_1cdb_4160_a975_615f6a38bad1.slice/crio-ea4138ab041fcd6eca29c34bab08fd1e5de4fbef425059b2051e26b05e3a920c WatchSource:0}: Error finding container ea4138ab041fcd6eca29c34bab08fd1e5de4fbef425059b2051e26b05e3a920c: Status 404 returned error can't find the container with id ea4138ab041fcd6eca29c34bab08fd1e5de4fbef425059b2051e26b05e3a920c Sep 09 10:26:01.598761 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.598551 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 10:26:01.600072 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.600052 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5dpbc" Sep 09 10:26:01.601091 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:01.601070 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf313a4_2a86_47a9_8b41_934ec362c7a5.slice/crio-e250c247b49a8d5e2d6c905c4cd3186cc8dc6be08b3f1c42dc5d2312aef25f7e WatchSource:0}: Error finding container e250c247b49a8d5e2d6c905c4cd3186cc8dc6be08b3f1c42dc5d2312aef25f7e: Status 404 returned error can't find the container with id e250c247b49a8d5e2d6c905c4cd3186cc8dc6be08b3f1c42dc5d2312aef25f7e Sep 09 10:26:01.610798 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:01.610771 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6aee692_b2c3_48d5_af2b_90bdeb0a29a7.slice/crio-95777f3a014c8cca54ee66f455d140608e324b64159abec331fd85d66b7d5b4d WatchSource:0}: Error finding container 95777f3a014c8cca54ee66f455d140608e324b64159abec331fd85d66b7d5b4d: Status 404 returned error can't find the container with id 95777f3a014c8cca54ee66f455d140608e324b64159abec331fd85d66b7d5b4d Sep 09 10:26:01.621196 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.621168 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xzlmj" Sep 09 10:26:01.628923 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:01.628891 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode342f56b_7a5f_4175_bc85_285df46a3475.slice/crio-a91758cc894a49e93e3d06e2fc4565ccd0a5ce7fa70629b65b0f93366622fcb3 WatchSource:0}: Error finding container a91758cc894a49e93e3d06e2fc4565ccd0a5ce7fa70629b65b0f93366622fcb3: Status 404 returned error can't find the container with id a91758cc894a49e93e3d06e2fc4565ccd0a5ce7fa70629b65b0f93366622fcb3 Sep 09 10:26:01.637646 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.637623 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcg2k" Sep 09 10:26:01.641810 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.641779 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:01.645463 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:01.645433 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc90a57_0901_47af_9adf_a3f4f52e7d4a.slice/crio-8b960014aed3b3c5e0e878843cc35109c7eda5bb4a6a6b73ccf604369879cf08 WatchSource:0}: Error finding container 8b960014aed3b3c5e0e878843cc35109c7eda5bb4a6a6b73ccf604369879cf08: Status 404 returned error can't find the container with id 8b960014aed3b3c5e0e878843cc35109c7eda5bb4a6a6b73ccf604369879cf08 Sep 09 10:26:01.650551 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:01.650510 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a56b285_3f3c_4f6d_a939_8a95ae00e8da.slice/crio-ca65fc934a32486dc8c3b032a02e2b1c22c075b95fa4aa90085afc5c8536488e WatchSource:0}: Error finding container ca65fc934a32486dc8c3b032a02e2b1c22c075b95fa4aa90085afc5c8536488e: Status 404 returned error can't find the container with id ca65fc934a32486dc8c3b032a02e2b1c22c075b95fa4aa90085afc5c8536488e Sep 09 10:26:01.903915 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:01.903882 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcecf6a195bd7af1b2519a96aa95ed769.slice/crio-07c8f45135593a6e93250a6e0e3ed84a874777c2495fa847f4ffa27c49482d1e WatchSource:0}: Error finding container 07c8f45135593a6e93250a6e0e3ed84a874777c2495fa847f4ffa27c49482d1e: Status 404 returned error can't find the container with id 07c8f45135593a6e93250a6e0e3ed84a874777c2495fa847f4ffa27c49482d1e Sep 09 10:26:01.915068 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.915038 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b697c056-99a2-4c91-a7c3-1f787553973e-original-pull-secret\") pod \"global-pull-secret-syncer-k9fqd\" (UID: \"b697c056-99a2-4c91-a7c3-1f787553973e\") " pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:01.915160 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:01.915082 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs\") pod \"network-metrics-daemon-c4s48\" (UID: \"1b8cf871-4dea-4ac5-868a-35b02f455e65\") " pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:26:01.915197 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:01.915178 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 10:26:01.915197 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:01.915179 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 10:26:01.915257 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:01.915224 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs podName:1b8cf871-4dea-4ac5-868a-35b02f455e65 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:02.915212946 +0000 UTC m=+3.089815250 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs") pod "network-metrics-daemon-c4s48" (UID: "1b8cf871-4dea-4ac5-868a-35b02f455e65") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 10:26:01.915257 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:01.915237 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b697c056-99a2-4c91-a7c3-1f787553973e-original-pull-secret podName:b697c056-99a2-4c91-a7c3-1f787553973e nodeName:}" failed. No retries permitted until 2026-09-09 10:26:02.915231417 +0000 UTC m=+3.089833720 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b697c056-99a2-4c91-a7c3-1f787553973e-original-pull-secret") pod "global-pull-secret-syncer-k9fqd" (UID: "b697c056-99a2-4c91-a7c3-1f787553973e") : object "kube-system"/"original-pull-secret" not registered Sep 09 10:26:02.015714 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:02.015675 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54scz\" (UniqueName: \"kubernetes.io/projected/946804f0-f29b-4f4c-8990-3cc0bf40c79e-kube-api-access-54scz\") pod \"network-check-target-vsngp\" (UID: \"946804f0-f29b-4f4c-8990-3cc0bf40c79e\") " pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:26:02.015866 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:02.015850 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 10:26:02.015923 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:02.015876 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 10:26:02.015923 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:02.015887 2559 projected.go:196] Error preparing data for projected volume kube-api-access-54scz for pod openshift-network-diagnostics/network-check-target-vsngp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 10:26:02.015982 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:02.015941 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/946804f0-f29b-4f4c-8990-3cc0bf40c79e-kube-api-access-54scz podName:946804f0-f29b-4f4c-8990-3cc0bf40c79e nodeName:}" failed. No retries permitted until 2026-09-09 10:26:03.015927528 +0000 UTC m=+3.190529836 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-54scz" (UniqueName: "kubernetes.io/projected/946804f0-f29b-4f4c-8990-3cc0bf40c79e-kube-api-access-54scz") pod "network-check-target-vsngp" (UID: "946804f0-f29b-4f4c-8990-3cc0bf40c79e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 10:26:02.048944 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:02.048913 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 10:26:02.278758 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:02.278597 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 10:21:01 +0000 UTC" deadline="2028-06-06 07:18:32.58972665 +0000 UTC" Sep 09 10:26:02.278758 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:02.278659 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15260h52m30.311072128s" Sep 09 10:26:02.334747 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:02.334683 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5dpbc" event={"ID":"f6aee692-b2c3-48d5-af2b-90bdeb0a29a7","Type":"ContainerStarted","Data":"95777f3a014c8cca54ee66f455d140608e324b64159abec331fd85d66b7d5b4d"} Sep 09 10:26:02.336664 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:02.336630 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-msgr4" event={"ID":"9ea95c24-1cdb-4160-a975-615f6a38bad1","Type":"ContainerStarted","Data":"ea4138ab041fcd6eca29c34bab08fd1e5de4fbef425059b2051e26b05e3a920c"} Sep 09 10:26:02.338558 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:02.338527 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4lsnr" event={"ID":"13c59e77-04fc-4549-8402-df18b180161f","Type":"ContainerStarted","Data":"6cb3cb34e377cb39d6d4370354da267a6d7c13aa64c3876ea69636d124ee7830"} Sep 09 10:26:02.341356 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:02.341329 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l9ttv" event={"ID":"3ceff104-498a-49b9-a2b1-0d9de58ebca3","Type":"ContainerStarted","Data":"d8d02e7cfc7ff4f36748339e4c9501b1f1d25d91b28ce89d93242a5b30016277"} Sep 09 10:26:02.348219 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:02.343826 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-133.ec2.internal" event={"ID":"cecf6a195bd7af1b2519a96aa95ed769","Type":"ContainerStarted","Data":"07c8f45135593a6e93250a6e0e3ed84a874777c2495fa847f4ffa27c49482d1e"} Sep 09 10:26:02.350887 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:02.350844 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xzlmj" event={"ID":"e342f56b-7a5f-4175-bc85-285df46a3475","Type":"ContainerStarted","Data":"a91758cc894a49e93e3d06e2fc4565ccd0a5ce7fa70629b65b0f93366622fcb3"} Sep 09 10:26:02.352916 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:02.352875 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-88lrx" event={"ID":"1bf313a4-2a86-47a9-8b41-934ec362c7a5","Type":"ContainerStarted","Data":"e250c247b49a8d5e2d6c905c4cd3186cc8dc6be08b3f1c42dc5d2312aef25f7e"} Sep 09 10:26:02.354772 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:02.354745 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bq9w9" event={"ID":"784872fd-624e-4323-862a-f1a040222214","Type":"ContainerStarted","Data":"35be6f5e09c85640e3e6237a163b8ac7afc8400d16739ddc6ef620ae8cdd9f6f"} Sep 09 10:26:02.357543 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:02.357514 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-133.ec2.internal" event={"ID":"03c4761a17877d2c4ef55ddef5cde326","Type":"ContainerStarted","Data":"b330b8cdf98cc60f0ce6274f9ec6752347deda720610e469706636e48d2b8554"} Sep 09 10:26:02.360295 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:02.360268 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" event={"ID":"0a56b285-3f3c-4f6d-a939-8a95ae00e8da","Type":"ContainerStarted","Data":"ca65fc934a32486dc8c3b032a02e2b1c22c075b95fa4aa90085afc5c8536488e"} Sep 09 10:26:02.364569 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:02.364534 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcg2k" event={"ID":"6bc90a57-0901-47af-9adf-a3f4f52e7d4a","Type":"ContainerStarted","Data":"8b960014aed3b3c5e0e878843cc35109c7eda5bb4a6a6b73ccf604369879cf08"} Sep 09 10:26:02.923765 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:02.923724 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b697c056-99a2-4c91-a7c3-1f787553973e-original-pull-secret\") pod \"global-pull-secret-syncer-k9fqd\" (UID: \"b697c056-99a2-4c91-a7c3-1f787553973e\") " pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:02.923979 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:02.923782 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs\") pod \"network-metrics-daemon-c4s48\" (UID: \"1b8cf871-4dea-4ac5-868a-35b02f455e65\") " pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:26:02.923979 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:02.923885 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 10:26:02.923979 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:02.923952 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs podName:1b8cf871-4dea-4ac5-868a-35b02f455e65 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:04.923933187 +0000 UTC m=+5.098535496 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs") pod "network-metrics-daemon-c4s48" (UID: "1b8cf871-4dea-4ac5-868a-35b02f455e65") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 10:26:02.924398 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:02.924377 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 10:26:02.924485 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:02.924472 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b697c056-99a2-4c91-a7c3-1f787553973e-original-pull-secret podName:b697c056-99a2-4c91-a7c3-1f787553973e nodeName:}" failed. No retries permitted until 2026-09-09 10:26:04.924454778 +0000 UTC m=+5.099057087 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b697c056-99a2-4c91-a7c3-1f787553973e-original-pull-secret") pod "global-pull-secret-syncer-k9fqd" (UID: "b697c056-99a2-4c91-a7c3-1f787553973e") : object "kube-system"/"original-pull-secret" not registered Sep 09 10:26:03.024132 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:03.024086 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54scz\" (UniqueName: \"kubernetes.io/projected/946804f0-f29b-4f4c-8990-3cc0bf40c79e-kube-api-access-54scz\") pod \"network-check-target-vsngp\" (UID: \"946804f0-f29b-4f4c-8990-3cc0bf40c79e\") " pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:26:03.024420 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:03.024343 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 10:26:03.024420 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:03.024365 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 10:26:03.024420 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:03.024378 2559 projected.go:196] Error preparing data for projected volume kube-api-access-54scz for pod openshift-network-diagnostics/network-check-target-vsngp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 10:26:03.024613 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:03.024437 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/946804f0-f29b-4f4c-8990-3cc0bf40c79e-kube-api-access-54scz podName:946804f0-f29b-4f4c-8990-3cc0bf40c79e nodeName:}" failed. No retries permitted until 2026-09-09 10:26:05.024420029 +0000 UTC m=+5.199022335 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-54scz" (UniqueName: "kubernetes.io/projected/946804f0-f29b-4f4c-8990-3cc0bf40c79e-kube-api-access-54scz") pod "network-check-target-vsngp" (UID: "946804f0-f29b-4f4c-8990-3cc0bf40c79e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 10:26:03.279162 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:03.279043 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 10:21:01 +0000 UTC" deadline="2028-05-24 03:15:52.216170114 +0000 UTC" Sep 09 10:26:03.279162 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:03.279110 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14944h49m48.937063755s" Sep 09 10:26:03.329139 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:03.328266 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:26:03.329139 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:03.328396 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsngp" podUID="946804f0-f29b-4f4c-8990-3cc0bf40c79e" Sep 09 10:26:03.329139 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:03.328843 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:03.329139 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:03.328942 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9fqd" podUID="b697c056-99a2-4c91-a7c3-1f787553973e" Sep 09 10:26:03.329139 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:03.329010 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:26:03.329139 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:03.329093 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4s48" podUID="1b8cf871-4dea-4ac5-868a-35b02f455e65" Sep 09 10:26:04.082655 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:04.082393 2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 10:26:04.942846 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:04.942778 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs\") pod \"network-metrics-daemon-c4s48\" (UID: \"1b8cf871-4dea-4ac5-868a-35b02f455e65\") " pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:26:04.943314 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:04.942872 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b697c056-99a2-4c91-a7c3-1f787553973e-original-pull-secret\") pod \"global-pull-secret-syncer-k9fqd\" (UID: \"b697c056-99a2-4c91-a7c3-1f787553973e\") " pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:04.943314 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:04.942998 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 10:26:04.943314 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:04.943064 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b697c056-99a2-4c91-a7c3-1f787553973e-original-pull-secret podName:b697c056-99a2-4c91-a7c3-1f787553973e nodeName:}" failed. No retries permitted until 2026-09-09 10:26:08.943043401 +0000 UTC m=+9.117645711 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b697c056-99a2-4c91-a7c3-1f787553973e-original-pull-secret") pod "global-pull-secret-syncer-k9fqd" (UID: "b697c056-99a2-4c91-a7c3-1f787553973e") : object "kube-system"/"original-pull-secret" not registered Sep 09 10:26:04.943314 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:04.943133 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 10:26:04.943314 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:04.943182 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs podName:1b8cf871-4dea-4ac5-868a-35b02f455e65 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:08.943170545 +0000 UTC m=+9.117772865 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs") pod "network-metrics-daemon-c4s48" (UID: "1b8cf871-4dea-4ac5-868a-35b02f455e65") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 10:26:05.044526 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:05.044276 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54scz\" (UniqueName: \"kubernetes.io/projected/946804f0-f29b-4f4c-8990-3cc0bf40c79e-kube-api-access-54scz\") pod \"network-check-target-vsngp\" (UID: \"946804f0-f29b-4f4c-8990-3cc0bf40c79e\") " pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:26:05.044526 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:05.044442 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 10:26:05.044526 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:05.044462 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 10:26:05.044526 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:05.044475 2559 projected.go:196] Error preparing data for projected volume kube-api-access-54scz for pod openshift-network-diagnostics/network-check-target-vsngp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 10:26:05.044854 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:05.044566 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/946804f0-f29b-4f4c-8990-3cc0bf40c79e-kube-api-access-54scz podName:946804f0-f29b-4f4c-8990-3cc0bf40c79e nodeName:}" failed. No retries permitted until 2026-09-09 10:26:09.04454809 +0000 UTC m=+9.219150407 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-54scz" (UniqueName: "kubernetes.io/projected/946804f0-f29b-4f4c-8990-3cc0bf40c79e-kube-api-access-54scz") pod "network-check-target-vsngp" (UID: "946804f0-f29b-4f4c-8990-3cc0bf40c79e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 10:26:05.328377 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:05.328290 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:26:05.328556 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:05.328419 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:05.328556 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:05.328431 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsngp" podUID="946804f0-f29b-4f4c-8990-3cc0bf40c79e" Sep 09 10:26:05.328674 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:05.328569 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9fqd" podUID="b697c056-99a2-4c91-a7c3-1f787553973e" Sep 09 10:26:05.328674 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:05.328618 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:26:05.328740 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:05.328671 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4s48" podUID="1b8cf871-4dea-4ac5-868a-35b02f455e65" Sep 09 10:26:07.328794 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:07.328233 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:07.328794 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:07.328234 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:26:07.328794 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:07.328395 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9fqd" podUID="b697c056-99a2-4c91-a7c3-1f787553973e" Sep 09 10:26:07.328794 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:07.328416 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:26:07.328794 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:07.328563 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4s48" podUID="1b8cf871-4dea-4ac5-868a-35b02f455e65" Sep 09 10:26:07.328794 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:07.328636 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsngp" podUID="946804f0-f29b-4f4c-8990-3cc0bf40c79e" Sep 09 10:26:08.978826 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:08.978785 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b697c056-99a2-4c91-a7c3-1f787553973e-original-pull-secret\") pod \"global-pull-secret-syncer-k9fqd\" (UID: \"b697c056-99a2-4c91-a7c3-1f787553973e\") " pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:08.979367 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:08.978855 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs\") pod \"network-metrics-daemon-c4s48\" (UID: \"1b8cf871-4dea-4ac5-868a-35b02f455e65\") " pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:26:08.979367 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:08.978971 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 10:26:08.979367 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:08.979008 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 10:26:08.979367 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:08.979049 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b697c056-99a2-4c91-a7c3-1f787553973e-original-pull-secret podName:b697c056-99a2-4c91-a7c3-1f787553973e nodeName:}" failed. No retries permitted until 2026-09-09 10:26:16.979028025 +0000 UTC m=+17.153630351 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b697c056-99a2-4c91-a7c3-1f787553973e-original-pull-secret") pod "global-pull-secret-syncer-k9fqd" (UID: "b697c056-99a2-4c91-a7c3-1f787553973e") : object "kube-system"/"original-pull-secret" not registered Sep 09 10:26:08.979367 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:08.979071 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs podName:1b8cf871-4dea-4ac5-868a-35b02f455e65 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:16.979060601 +0000 UTC m=+17.153662909 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs") pod "network-metrics-daemon-c4s48" (UID: "1b8cf871-4dea-4ac5-868a-35b02f455e65") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 10:26:09.079769 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:09.079728 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54scz\" (UniqueName: \"kubernetes.io/projected/946804f0-f29b-4f4c-8990-3cc0bf40c79e-kube-api-access-54scz\") pod \"network-check-target-vsngp\" (UID: \"946804f0-f29b-4f4c-8990-3cc0bf40c79e\") " pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:26:09.079968 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:09.079872 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 10:26:09.079968 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:09.079900 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 10:26:09.079968 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:09.079913 2559 projected.go:196] Error preparing data for projected volume kube-api-access-54scz for pod openshift-network-diagnostics/network-check-target-vsngp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 10:26:09.080145 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:09.080021 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/946804f0-f29b-4f4c-8990-3cc0bf40c79e-kube-api-access-54scz podName:946804f0-f29b-4f4c-8990-3cc0bf40c79e nodeName:}" failed. No retries permitted until 2026-09-09 10:26:17.079998405 +0000 UTC m=+17.254600709 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-54scz" (UniqueName: "kubernetes.io/projected/946804f0-f29b-4f4c-8990-3cc0bf40c79e-kube-api-access-54scz") pod "network-check-target-vsngp" (UID: "946804f0-f29b-4f4c-8990-3cc0bf40c79e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 10:26:09.328476 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:09.328443 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:26:09.328698 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:09.328491 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:09.328698 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:09.328623 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4s48" podUID="1b8cf871-4dea-4ac5-868a-35b02f455e65" Sep 09 10:26:09.328698 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:09.328686 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:26:09.328925 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:09.328749 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsngp" podUID="946804f0-f29b-4f4c-8990-3cc0bf40c79e" Sep 09 10:26:09.328925 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:09.328677 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9fqd" podUID="b697c056-99a2-4c91-a7c3-1f787553973e" Sep 09 10:26:11.328281 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:11.328243 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:11.328806 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:11.328291 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:26:11.328806 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:11.328291 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:26:11.328806 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:11.328369 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9fqd" podUID="b697c056-99a2-4c91-a7c3-1f787553973e" Sep 09 10:26:11.328806 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:11.328458 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4s48" podUID="1b8cf871-4dea-4ac5-868a-35b02f455e65" Sep 09 10:26:11.328806 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:11.328525 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsngp" podUID="946804f0-f29b-4f4c-8990-3cc0bf40c79e" Sep 09 10:26:13.328305 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:13.328265 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:26:13.328771 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:13.328381 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4s48" podUID="1b8cf871-4dea-4ac5-868a-35b02f455e65" Sep 09 10:26:13.328771 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:13.328281 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:26:13.328771 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:13.328450 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsngp" podUID="946804f0-f29b-4f4c-8990-3cc0bf40c79e" Sep 09 10:26:13.328771 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:13.328266 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:13.328771 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:13.328518 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9fqd" podUID="b697c056-99a2-4c91-a7c3-1f787553973e" Sep 09 10:26:15.328241 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:15.328204 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:26:15.328241 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:15.328225 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:26:15.328921 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:15.328205 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:15.328921 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:15.328357 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4s48" podUID="1b8cf871-4dea-4ac5-868a-35b02f455e65" Sep 09 10:26:15.328921 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:15.328485 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9fqd" podUID="b697c056-99a2-4c91-a7c3-1f787553973e" Sep 09 10:26:15.328921 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:15.328553 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsngp" podUID="946804f0-f29b-4f4c-8990-3cc0bf40c79e" Sep 09 10:26:17.038553 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:17.038442 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b697c056-99a2-4c91-a7c3-1f787553973e-original-pull-secret\") pod \"global-pull-secret-syncer-k9fqd\" (UID: \"b697c056-99a2-4c91-a7c3-1f787553973e\") " pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:17.038553 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:17.038518 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs\") pod \"network-metrics-daemon-c4s48\" (UID: \"1b8cf871-4dea-4ac5-868a-35b02f455e65\") " pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:26:17.038996 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:17.038621 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 10:26:17.038996 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:17.038681 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs podName:1b8cf871-4dea-4ac5-868a-35b02f455e65 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:33.038666525 +0000 UTC m=+33.213268833 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs") pod "network-metrics-daemon-c4s48" (UID: "1b8cf871-4dea-4ac5-868a-35b02f455e65") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 10:26:17.038996 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:17.038622 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 10:26:17.038996 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:17.038771 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b697c056-99a2-4c91-a7c3-1f787553973e-original-pull-secret podName:b697c056-99a2-4c91-a7c3-1f787553973e nodeName:}" failed. No retries permitted until 2026-09-09 10:26:33.038755446 +0000 UTC m=+33.213357748 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b697c056-99a2-4c91-a7c3-1f787553973e-original-pull-secret") pod "global-pull-secret-syncer-k9fqd" (UID: "b697c056-99a2-4c91-a7c3-1f787553973e") : object "kube-system"/"original-pull-secret" not registered Sep 09 10:26:17.139647 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:17.139609 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54scz\" (UniqueName: \"kubernetes.io/projected/946804f0-f29b-4f4c-8990-3cc0bf40c79e-kube-api-access-54scz\") pod \"network-check-target-vsngp\" (UID: \"946804f0-f29b-4f4c-8990-3cc0bf40c79e\") " pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:26:17.139825 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:17.139757 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 10:26:17.139825 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:17.139780 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 10:26:17.139825 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:17.139792 2559 projected.go:196] Error preparing data for projected volume kube-api-access-54scz for pod openshift-network-diagnostics/network-check-target-vsngp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 10:26:17.139967 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:17.139854 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/946804f0-f29b-4f4c-8990-3cc0bf40c79e-kube-api-access-54scz podName:946804f0-f29b-4f4c-8990-3cc0bf40c79e nodeName:}" failed. No retries permitted until 2026-09-09 10:26:33.139836761 +0000 UTC m=+33.314439124 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-54scz" (UniqueName: "kubernetes.io/projected/946804f0-f29b-4f4c-8990-3cc0bf40c79e-kube-api-access-54scz") pod "network-check-target-vsngp" (UID: "946804f0-f29b-4f4c-8990-3cc0bf40c79e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 10:26:17.328541 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:17.328494 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:26:17.328739 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:17.328554 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:26:17.328739 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:17.328562 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:17.328739 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:17.328636 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsngp" podUID="946804f0-f29b-4f4c-8990-3cc0bf40c79e" Sep 09 10:26:17.328876 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:17.328808 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4s48" podUID="1b8cf871-4dea-4ac5-868a-35b02f455e65" Sep 09 10:26:17.328935 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:17.328894 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9fqd" podUID="b697c056-99a2-4c91-a7c3-1f787553973e" Sep 09 10:26:19.328751 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:19.328713 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:26:19.329239 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:19.328756 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:19.329239 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:19.328756 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:26:19.329239 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:19.328847 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsngp" podUID="946804f0-f29b-4f4c-8990-3cc0bf40c79e" Sep 09 10:26:19.329239 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:19.328909 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9fqd" podUID="b697c056-99a2-4c91-a7c3-1f787553973e" Sep 09 10:26:19.329239 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:19.329010 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4s48" podUID="1b8cf871-4dea-4ac5-868a-35b02f455e65" Sep 09 10:26:21.328882 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:21.328673 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:26:21.329365 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:21.328707 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:21.329365 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:21.328972 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsngp" podUID="946804f0-f29b-4f4c-8990-3cc0bf40c79e" Sep 09 10:26:21.329365 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:21.328721 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:26:21.329365 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:21.329117 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4s48" podUID="1b8cf871-4dea-4ac5-868a-35b02f455e65" Sep 09 10:26:21.329365 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:21.329018 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9fqd" podUID="b697c056-99a2-4c91-a7c3-1f787553973e" Sep 09 10:26:21.420104 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:21.420076 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d22ff_0a56b285-3f3c-4f6d-a939-8a95ae00e8da/ovn-acl-logging/0.log" Sep 09 10:26:21.422137 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:21.421995 2559 generic.go:356] "Generic (PLEG): container finished" podID="0a56b285-3f3c-4f6d-a939-8a95ae00e8da" containerID="2b2b5462d345b700f3738557353d9f00314c4e26df013babd35e3e888eeb9c4a" exitCode=1 Sep 09 10:26:21.422137 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:21.422116 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" event={"ID":"0a56b285-3f3c-4f6d-a939-8a95ae00e8da","Type":"ContainerStarted","Data":"ae3e44098f3cb6aea7262be783e530e8080ef362d38efb0a534c78b23636b578"} Sep 09 10:26:21.422350 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:21.422201 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" event={"ID":"0a56b285-3f3c-4f6d-a939-8a95ae00e8da","Type":"ContainerStarted","Data":"e3bdeffef88279b0238ad7c39fa45b4d2604426951dda65db9e8f8b8d19b67c2"} Sep 09 10:26:21.422350 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:21.422267 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" event={"ID":"0a56b285-3f3c-4f6d-a939-8a95ae00e8da","Type":"ContainerDied","Data":"2b2b5462d345b700f3738557353d9f00314c4e26df013babd35e3e888eeb9c4a"} Sep 09 10:26:21.422350 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:21.422322 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" event={"ID":"0a56b285-3f3c-4f6d-a939-8a95ae00e8da","Type":"ContainerStarted","Data":"6142f114900620e0aadb659f03d87261ef8121a572428a1f1fd83d00c630b157"} Sep 09 10:26:21.427594 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:21.427491 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-msgr4" event={"ID":"9ea95c24-1cdb-4160-a975-615f6a38bad1","Type":"ContainerStarted","Data":"39c0639248a49201e846748e40b0840c8e49a5d4e88a3a2b30de88cfc10bcffc"} Sep 09 10:26:21.432791 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:21.432732 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-133.ec2.internal" event={"ID":"cecf6a195bd7af1b2519a96aa95ed769","Type":"ContainerStarted","Data":"547ce459a6b615ba99606bb8cc88fdaacf553ffd9ed8966409b70ff556b1c531"} Sep 09 10:26:21.466197 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:21.466122 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-msgr4" podStartSLOduration=2.358682987 podStartE2EDuration="21.46609932s" totalImagesPullingTime="19.107416333s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:21.45217652 +0000 UTC m=+21.626778846" watchObservedRunningTime="2026-09-09 10:26:21.46609932 +0000 UTC m=+21.640701647" Sep 09 10:26:21.467156 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:21.467114 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-133.ec2.internal" podStartSLOduration=20.467099323 podStartE2EDuration="20.467099323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:21.466272544 +0000 UTC m=+21.640874859" watchObservedRunningTime="2026-09-09 10:26:21.467099323 +0000 UTC m=+21.641701649" Sep 09 10:26:22.436716 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:22.436449 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-88lrx" event={"ID":"1bf313a4-2a86-47a9-8b41-934ec362c7a5","Type":"ContainerStarted","Data":"b013d5caf0c5514e796fd4c649a87e329d016fd7cc6777c85dfedcea3fc6faf6"} Sep 09 10:26:22.437933 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:22.437906 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bq9w9" event={"ID":"784872fd-624e-4323-862a-f1a040222214","Type":"ContainerStarted","Data":"028fc2bcc9f93ba484cbdf3e4c7016a45fc912b5f5364b8fe7c153b6e387e0f5"} Sep 09 10:26:22.439444 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:22.439417 2559 generic.go:356] "Generic (PLEG): container finished" podID="03c4761a17877d2c4ef55ddef5cde326" containerID="4fed0d389763c3922f0629dce2e16bd6fe7b68d4e00e5dce520cd2ffefef4814" exitCode=0 Sep 09 10:26:22.439641 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:22.439618 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-133.ec2.internal" event={"ID":"03c4761a17877d2c4ef55ddef5cde326","Type":"ContainerDied","Data":"4fed0d389763c3922f0629dce2e16bd6fe7b68d4e00e5dce520cd2ffefef4814"} Sep 09 10:26:22.442420 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:22.442404 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d22ff_0a56b285-3f3c-4f6d-a939-8a95ae00e8da/ovn-acl-logging/0.log" Sep 09 10:26:22.442801 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:22.442784 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" event={"ID":"0a56b285-3f3c-4f6d-a939-8a95ae00e8da","Type":"ContainerStarted","Data":"798a0b2ebfcfcdef79af40bd7e185ba6121f6e9897ba53a115a30d9c3d20710a"} Sep 09 10:26:22.442863 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:22.442850 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" event={"ID":"0a56b285-3f3c-4f6d-a939-8a95ae00e8da","Type":"ContainerStarted","Data":"9b8f1a0523c297ed0c65a594cf9d2726f77bdd76167626e8769e72872a19d30f"} Sep 09 10:26:22.444204 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:22.444176 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcg2k" event={"ID":"6bc90a57-0901-47af-9adf-a3f4f52e7d4a","Type":"ContainerStarted","Data":"996041280acb67be28275bd467b238a24710cad9c59eac95d39474a3cfa7e20a"} Sep 09 10:26:22.445669 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:22.445646 2559 generic.go:356] "Generic (PLEG): container finished" podID="f6aee692-b2c3-48d5-af2b-90bdeb0a29a7" containerID="168704bab812b482af5a65356a77785afdd32d4f8d3211976403069f73099b36" exitCode=0 Sep 09 10:26:22.445745 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:22.445721 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5dpbc" event={"ID":"f6aee692-b2c3-48d5-af2b-90bdeb0a29a7","Type":"ContainerDied","Data":"168704bab812b482af5a65356a77785afdd32d4f8d3211976403069f73099b36"} Sep 09 10:26:22.447115 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:22.447038 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4lsnr" event={"ID":"13c59e77-04fc-4549-8402-df18b180161f","Type":"ContainerStarted","Data":"5eec8c64c44fe3097572d14bbc7790a3ce7f24ec862153920e539b0972ed7660"} Sep 09 10:26:22.448430 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:22.448413 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l9ttv" event={"ID":"3ceff104-498a-49b9-a2b1-0d9de58ebca3","Type":"ContainerStarted","Data":"65d5f0e680c551006804aac18484ebab7dafb6dd607a0903502ec5cde8b6ca4a"} Sep 09 10:26:22.449928 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:22.449906 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xzlmj" event={"ID":"e342f56b-7a5f-4175-bc85-285df46a3475","Type":"ContainerStarted","Data":"a395349d920f7c407bfbc2f2a5716406875dcd98e3480b78d1868f49eabc41b0"} Sep 09 10:26:22.464894 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:22.464849 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-88lrx" podStartSLOduration=2.7474528449999998 podStartE2EDuration="22.46483336s" totalImagesPullingTime="19.717380515s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:22.464725321 +0000 UTC m=+22.639327637" watchObservedRunningTime="2026-09-09 10:26:22.46483336 +0000 UTC m=+22.639435684" Sep 09 10:26:22.485840 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:22.485782 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-l9ttv" podStartSLOduration=3.380559964 podStartE2EDuration="22.485758655s" totalImagesPullingTime="19.105198691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:22.485205896 +0000 UTC m=+22.659808223" watchObservedRunningTime="2026-09-09 10:26:22.485758655 +0000 UTC m=+22.660360981" Sep 09 10:26:22.503103 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:22.503048 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bq9w9" podStartSLOduration=3.461891098 podStartE2EDuration="22.503030111s" totalImagesPullingTime="19.041139013s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:22.502571263 +0000 UTC m=+22.677173589" watchObservedRunningTime="2026-09-09 10:26:22.503030111 +0000 UTC m=+22.677632467" Sep 09 10:26:22.541388 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:22.541334 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xzlmj" podStartSLOduration=3.467022986 podStartE2EDuration="22.541316845s" totalImagesPullingTime="19.074293859s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:22.519789734 +0000 UTC m=+22.694392058" watchObservedRunningTime="2026-09-09 10:26:22.541316845 +0000 UTC m=+22.715919169" Sep 09 10:26:22.564331 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:22.564277 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4lsnr" podStartSLOduration=3.501980633 podStartE2EDuration="22.564260886s" totalImagesPullingTime="19.062280253s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:22.564209061 +0000 UTC m=+22.738811390" watchObservedRunningTime="2026-09-09 10:26:22.564260886 +0000 UTC m=+22.738863211" Sep 09 10:26:23.072676 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:23.072475 2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 09 10:26:23.121106 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:23.121062 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-l9ttv" Sep 09 10:26:23.121789 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:23.121766 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-l9ttv" Sep 09 10:26:23.276326 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:23.276070 2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T10:26:23.072669494Z","UUID":"405791c3-902f-41a0-8bab-77e0712f90d0","Handler":null,"Name":"","Endpoint":""} Sep 09 10:26:23.277643 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:23.277622 2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 09 10:26:23.277753 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:23.277653 2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 09 10:26:23.328157 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:23.328127 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:23.328409 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:23.328161 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:26:23.328409 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:23.328161 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:26:23.328409 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:23.328227 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9fqd" podUID="b697c056-99a2-4c91-a7c3-1f787553973e" Sep 09 10:26:23.328409 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:23.328331 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsngp" podUID="946804f0-f29b-4f4c-8990-3cc0bf40c79e" Sep 09 10:26:23.328409 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:23.328405 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4s48" podUID="1b8cf871-4dea-4ac5-868a-35b02f455e65" Sep 09 10:26:23.453565 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:23.453523 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcg2k" event={"ID":"6bc90a57-0901-47af-9adf-a3f4f52e7d4a","Type":"ContainerStarted","Data":"32418116c395d0fdfb86cead0c46dce201f735e8369e3e4cf22db9c4807fe62f"} Sep 09 10:26:23.455350 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:23.455318 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-133.ec2.internal" event={"ID":"03c4761a17877d2c4ef55ddef5cde326","Type":"ContainerStarted","Data":"1f6f3bbc5ef5e7c673b6c0be8fcad0cbb49f0b6e9937857d1a60a0bd05f79fa8"} Sep 09 10:26:23.456102 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:23.456081 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-l9ttv" Sep 09 10:26:23.456290 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:23.456272 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l9ttv" Sep 09 10:26:23.475424 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:23.475376 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-133.ec2.internal" podStartSLOduration=22.475362024 podStartE2EDuration="22.475362024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:23.475023283 +0000 UTC m=+23.649625607" watchObservedRunningTime="2026-09-09 10:26:23.475362024 +0000 UTC m=+23.649964349" Sep 09 10:26:24.461008 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:24.460979 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d22ff_0a56b285-3f3c-4f6d-a939-8a95ae00e8da/ovn-acl-logging/0.log" Sep 09 10:26:24.461432 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:24.461368 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" event={"ID":"0a56b285-3f3c-4f6d-a939-8a95ae00e8da","Type":"ContainerStarted","Data":"8a2a8ed24fb305906704a5516ad880705c047583e9829aa9c81364df59a182cb"} Sep 09 10:26:24.463421 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:24.463394 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcg2k" event={"ID":"6bc90a57-0901-47af-9adf-a3f4f52e7d4a","Type":"ContainerStarted","Data":"8b2fe4c014678f01bb8b31b35d0406554c75d18c0a699700167e8d2b869f854d"} Sep 09 10:26:24.487334 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:24.487252 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcg2k" podStartSLOduration=3.175785639 podStartE2EDuration="24.487237515s" totalImagesPullingTime="21.311451876s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:26:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:24.487049612 +0000 UTC m=+24.661651937" watchObservedRunningTime="2026-09-09 10:26:24.487237515 +0000 UTC m=+24.661839839" Sep 09 10:26:25.328312 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:25.328280 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:26:25.328473 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:25.328312 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:26:25.328473 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:25.328312 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:25.328473 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:25.328407 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4s48" podUID="1b8cf871-4dea-4ac5-868a-35b02f455e65" Sep 09 10:26:25.328473 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:25.328459 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsngp" podUID="946804f0-f29b-4f4c-8990-3cc0bf40c79e" Sep 09 10:26:25.328663 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:25.328554 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9fqd" podUID="b697c056-99a2-4c91-a7c3-1f787553973e" Sep 09 10:26:26.471545 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:26.471297 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d22ff_0a56b285-3f3c-4f6d-a939-8a95ae00e8da/ovn-acl-logging/0.log" Sep 09 10:26:26.471938 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:26.471910 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" event={"ID":"0a56b285-3f3c-4f6d-a939-8a95ae00e8da","Type":"ContainerStarted","Data":"2c4ea674ee63bf1d1ec9acc964a3dc98fbcd426d60ee4328677814198683ff0c"} Sep 09 10:26:26.472219 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:26.472198 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:26.472305 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:26.472274 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:26.472455 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:26.472438 2559 scope.go:117] "RemoveContainer" containerID="2b2b5462d345b700f3738557353d9f00314c4e26df013babd35e3e888eeb9c4a" Sep 09 10:26:26.489407 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:26.489379 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:27.328253 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:27.328220 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:27.328411 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:27.328263 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:26:27.328411 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:27.328227 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:26:27.328411 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:27.328336 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9fqd" podUID="b697c056-99a2-4c91-a7c3-1f787553973e" Sep 09 10:26:27.328557 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:27.328418 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsngp" podUID="946804f0-f29b-4f4c-8990-3cc0bf40c79e" Sep 09 10:26:27.328557 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:27.328541 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4s48" podUID="1b8cf871-4dea-4ac5-868a-35b02f455e65" Sep 09 10:26:28.165442 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:28.165407 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-k9fqd"] Sep 09 10:26:28.166111 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:28.165541 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:28.166111 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:28.165658 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9fqd" podUID="b697c056-99a2-4c91-a7c3-1f787553973e" Sep 09 10:26:28.169546 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:28.169483 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vsngp"] Sep 09 10:26:28.169670 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:28.169644 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:26:28.169777 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:28.169753 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsngp" podUID="946804f0-f29b-4f4c-8990-3cc0bf40c79e" Sep 09 10:26:28.170316 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:28.170292 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c4s48"] Sep 09 10:26:28.170436 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:28.170416 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:26:28.170569 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:28.170545 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4s48" podUID="1b8cf871-4dea-4ac5-868a-35b02f455e65" Sep 09 10:26:28.482742 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:28.482716 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d22ff_0a56b285-3f3c-4f6d-a939-8a95ae00e8da/ovn-acl-logging/0.log" Sep 09 10:26:28.483156 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:28.483128 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" event={"ID":"0a56b285-3f3c-4f6d-a939-8a95ae00e8da","Type":"ContainerStarted","Data":"3a35cc2f4bfade183fc882b9df373534d29460d54184ed9fb60e06c9e8c0a48f"} Sep 09 10:26:28.483542 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:28.483524 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:28.511030 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:28.511003 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:26:28.547854 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:28.547809 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" podStartSLOduration=9.459160014 podStartE2EDuration="28.547793076s" totalImagesPullingTime="19.088633062s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:28.517088616 +0000 UTC m=+28.691690940" watchObservedRunningTime="2026-09-09 10:26:28.547793076 +0000 UTC m=+28.722395392" Sep 09 10:26:29.487343 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:29.487308 2559 generic.go:356] "Generic (PLEG): container finished" podID="f6aee692-b2c3-48d5-af2b-90bdeb0a29a7" containerID="a9622faf2269b1b01c4f8a514486a3648873d1efb1c06f1dfd52a5be9154732a" exitCode=0 Sep 09 10:26:29.487894 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:29.487380 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5dpbc" event={"ID":"f6aee692-b2c3-48d5-af2b-90bdeb0a29a7","Type":"ContainerDied","Data":"a9622faf2269b1b01c4f8a514486a3648873d1efb1c06f1dfd52a5be9154732a"} Sep 09 10:26:30.329074 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:30.328885 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:30.329242 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:30.328976 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:26:30.329242 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:30.329141 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9fqd" podUID="b697c056-99a2-4c91-a7c3-1f787553973e" Sep 09 10:26:30.329356 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:30.329236 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4s48" podUID="1b8cf871-4dea-4ac5-868a-35b02f455e65" Sep 09 10:26:30.329356 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:30.329031 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:26:30.329356 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:30.329309 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsngp" podUID="946804f0-f29b-4f4c-8990-3cc0bf40c79e" Sep 09 10:26:31.494701 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:31.494670 2559 generic.go:356] "Generic (PLEG): container finished" podID="f6aee692-b2c3-48d5-af2b-90bdeb0a29a7" containerID="b152be787cb3a42ba9091476bc1e55558a6feaf9d709d6a8c98737bf74e889e8" exitCode=0 Sep 09 10:26:31.495079 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:31.494715 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5dpbc" event={"ID":"f6aee692-b2c3-48d5-af2b-90bdeb0a29a7","Type":"ContainerDied","Data":"b152be787cb3a42ba9091476bc1e55558a6feaf9d709d6a8c98737bf74e889e8"} Sep 09 10:26:32.328746 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:32.328709 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:26:32.328905 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:32.328754 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:32.328905 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:32.328815 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:26:32.328905 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:32.328827 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4s48" podUID="1b8cf871-4dea-4ac5-868a-35b02f455e65" Sep 09 10:26:32.329003 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:32.328898 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsngp" podUID="946804f0-f29b-4f4c-8990-3cc0bf40c79e" Sep 09 10:26:32.329003 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:32.328990 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9fqd" podUID="b697c056-99a2-4c91-a7c3-1f787553973e" Sep 09 10:26:33.055078 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:33.054982 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs\") pod \"network-metrics-daemon-c4s48\" (UID: \"1b8cf871-4dea-4ac5-868a-35b02f455e65\") " pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:26:33.055078 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:33.055039 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b697c056-99a2-4c91-a7c3-1f787553973e-original-pull-secret\") pod \"global-pull-secret-syncer-k9fqd\" (UID: \"b697c056-99a2-4c91-a7c3-1f787553973e\") " pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:33.055711 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:33.055132 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 10:26:33.055711 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:33.055182 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b697c056-99a2-4c91-a7c3-1f787553973e-original-pull-secret podName:b697c056-99a2-4c91-a7c3-1f787553973e nodeName:}" failed. No retries permitted until 2026-09-09 10:27:05.055168699 +0000 UTC m=+65.229771001 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b697c056-99a2-4c91-a7c3-1f787553973e-original-pull-secret") pod "global-pull-secret-syncer-k9fqd" (UID: "b697c056-99a2-4c91-a7c3-1f787553973e") : object "kube-system"/"original-pull-secret" not registered Sep 09 10:26:33.055711 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:33.055132 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 10:26:33.055711 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:33.055264 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs podName:1b8cf871-4dea-4ac5-868a-35b02f455e65 nodeName:}" failed. No retries permitted until 2026-09-09 10:27:05.055250716 +0000 UTC m=+65.229853022 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs") pod "network-metrics-daemon-c4s48" (UID: "1b8cf871-4dea-4ac5-868a-35b02f455e65") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 10:26:33.156088 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:33.156050 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54scz\" (UniqueName: \"kubernetes.io/projected/946804f0-f29b-4f4c-8990-3cc0bf40c79e-kube-api-access-54scz\") pod \"network-check-target-vsngp\" (UID: \"946804f0-f29b-4f4c-8990-3cc0bf40c79e\") " pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:26:33.156244 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:33.156192 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 10:26:33.156244 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:33.156205 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 10:26:33.156244 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:33.156214 2559 projected.go:196] Error preparing data for projected volume kube-api-access-54scz for pod openshift-network-diagnostics/network-check-target-vsngp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 10:26:33.156362 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:33.156260 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/946804f0-f29b-4f4c-8990-3cc0bf40c79e-kube-api-access-54scz podName:946804f0-f29b-4f4c-8990-3cc0bf40c79e nodeName:}" failed. No retries permitted until 2026-09-09 10:27:05.156248091 +0000 UTC m=+65.330850394 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-54scz" (UniqueName: "kubernetes.io/projected/946804f0-f29b-4f4c-8990-3cc0bf40c79e-kube-api-access-54scz") pod "network-check-target-vsngp" (UID: "946804f0-f29b-4f4c-8990-3cc0bf40c79e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 10:26:33.502590 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:33.502555 2559 generic.go:356] "Generic (PLEG): container finished" podID="f6aee692-b2c3-48d5-af2b-90bdeb0a29a7" containerID="b96bab589819c11792c5932c739316dfd2e21d47a1a7933f0d3909fbc7e8f3da" exitCode=0 Sep 09 10:26:33.502760 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:33.502609 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5dpbc" event={"ID":"f6aee692-b2c3-48d5-af2b-90bdeb0a29a7","Type":"ContainerDied","Data":"b96bab589819c11792c5932c739316dfd2e21d47a1a7933f0d3909fbc7e8f3da"} Sep 09 10:26:34.328042 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:34.328008 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:34.328364 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:34.328008 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:26:34.328364 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:34.328126 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9fqd" podUID="b697c056-99a2-4c91-a7c3-1f787553973e" Sep 09 10:26:34.328364 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:34.328225 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4s48" podUID="1b8cf871-4dea-4ac5-868a-35b02f455e65" Sep 09 10:26:34.328364 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:34.328020 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:26:34.328364 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:34.328318 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsngp" podUID="946804f0-f29b-4f4c-8990-3cc0bf40c79e" Sep 09 10:26:35.060799 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.060768 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-133.ec2.internal" event="NodeReady" Sep 09 10:26:35.061015 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.060898 2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 09 10:26:35.126791 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.126748 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-llvb2"] Sep 09 10:26:35.148892 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.148862 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jjw92"] Sep 09 10:26:35.149088 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.148979 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-llvb2" Sep 09 10:26:35.151966 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.151905 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 09 10:26:35.152603 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.152576 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 09 10:26:35.152720 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.152640 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 09 10:26:35.152720 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.152666 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wx2nx\"" Sep 09 10:26:35.158777 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.158749 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-llvb2"] Sep 09 10:26:35.158777 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.158781 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jjw92"] Sep 09 10:26:35.158980 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.158861 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jjw92" Sep 09 10:26:35.162090 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.162066 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-272r8\"" Sep 09 10:26:35.162373 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.162350 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 09 10:26:35.162664 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.162649 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 09 10:26:35.268773 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.268739 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmqkd\" (UniqueName: \"kubernetes.io/projected/88316610-e58d-4f66-ae6c-eb81674b4f43-kube-api-access-hmqkd\") pod \"dns-default-jjw92\" (UID: \"88316610-e58d-4f66-ae6c-eb81674b4f43\") " pod="openshift-dns/dns-default-jjw92" Sep 09 10:26:35.268773 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.268777 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/88316610-e58d-4f66-ae6c-eb81674b4f43-config-volume\") pod \"dns-default-jjw92\" (UID: \"88316610-e58d-4f66-ae6c-eb81674b4f43\") " pod="openshift-dns/dns-default-jjw92" Sep 09 10:26:35.269171 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.268793 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/88316610-e58d-4f66-ae6c-eb81674b4f43-metrics-tls\") pod \"dns-default-jjw92\" (UID: \"88316610-e58d-4f66-ae6c-eb81674b4f43\") " pod="openshift-dns/dns-default-jjw92" Sep 09 10:26:35.269171 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.268900 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88316610-e58d-4f66-ae6c-eb81674b4f43-tmp-dir\") pod \"dns-default-jjw92\" (UID: \"88316610-e58d-4f66-ae6c-eb81674b4f43\") " pod="openshift-dns/dns-default-jjw92" Sep 09 10:26:35.269171 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.268953 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53bd305a-c566-47e2-bbfe-cddf73635d77-cert\") pod \"ingress-canary-llvb2\" (UID: \"53bd305a-c566-47e2-bbfe-cddf73635d77\") " pod="openshift-ingress-canary/ingress-canary-llvb2" Sep 09 10:26:35.269171 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.269004 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk26p\" (UniqueName: \"kubernetes.io/projected/53bd305a-c566-47e2-bbfe-cddf73635d77-kube-api-access-rk26p\") pod \"ingress-canary-llvb2\" (UID: \"53bd305a-c566-47e2-bbfe-cddf73635d77\") " pod="openshift-ingress-canary/ingress-canary-llvb2" Sep 09 10:26:35.370234 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.370196 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53bd305a-c566-47e2-bbfe-cddf73635d77-cert\") pod \"ingress-canary-llvb2\" (UID: \"53bd305a-c566-47e2-bbfe-cddf73635d77\") " pod="openshift-ingress-canary/ingress-canary-llvb2" Sep 09 10:26:35.370736 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.370263 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk26p\" (UniqueName: \"kubernetes.io/projected/53bd305a-c566-47e2-bbfe-cddf73635d77-kube-api-access-rk26p\") pod \"ingress-canary-llvb2\" (UID: \"53bd305a-c566-47e2-bbfe-cddf73635d77\") " pod="openshift-ingress-canary/ingress-canary-llvb2" Sep 09 10:26:35.370736 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.370321 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmqkd\" (UniqueName: \"kubernetes.io/projected/88316610-e58d-4f66-ae6c-eb81674b4f43-kube-api-access-hmqkd\") pod \"dns-default-jjw92\" (UID: \"88316610-e58d-4f66-ae6c-eb81674b4f43\") " pod="openshift-dns/dns-default-jjw92" Sep 09 10:26:35.370736 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.370352 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/88316610-e58d-4f66-ae6c-eb81674b4f43-config-volume\") pod \"dns-default-jjw92\" (UID: \"88316610-e58d-4f66-ae6c-eb81674b4f43\") " pod="openshift-dns/dns-default-jjw92" Sep 09 10:26:35.370736 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.370376 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/88316610-e58d-4f66-ae6c-eb81674b4f43-metrics-tls\") pod \"dns-default-jjw92\" (UID: \"88316610-e58d-4f66-ae6c-eb81674b4f43\") " pod="openshift-dns/dns-default-jjw92" Sep 09 10:26:35.370736 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.370425 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88316610-e58d-4f66-ae6c-eb81674b4f43-tmp-dir\") pod \"dns-default-jjw92\" (UID: \"88316610-e58d-4f66-ae6c-eb81674b4f43\") " pod="openshift-dns/dns-default-jjw92" Sep 09 10:26:35.370736 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:35.370576 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 10:26:35.370736 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:35.370647 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88316610-e58d-4f66-ae6c-eb81674b4f43-metrics-tls podName:88316610-e58d-4f66-ae6c-eb81674b4f43 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:35.870626487 +0000 UTC m=+36.045228807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/88316610-e58d-4f66-ae6c-eb81674b4f43-metrics-tls") pod "dns-default-jjw92" (UID: "88316610-e58d-4f66-ae6c-eb81674b4f43") : secret "dns-default-metrics-tls" not found Sep 09 10:26:35.370736 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.370723 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88316610-e58d-4f66-ae6c-eb81674b4f43-tmp-dir\") pod \"dns-default-jjw92\" (UID: \"88316610-e58d-4f66-ae6c-eb81674b4f43\") " pod="openshift-dns/dns-default-jjw92" Sep 09 10:26:35.371121 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.370962 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/88316610-e58d-4f66-ae6c-eb81674b4f43-config-volume\") pod \"dns-default-jjw92\" (UID: \"88316610-e58d-4f66-ae6c-eb81674b4f43\") " pod="openshift-dns/dns-default-jjw92" Sep 09 10:26:35.375221 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.375196 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53bd305a-c566-47e2-bbfe-cddf73635d77-cert\") pod \"ingress-canary-llvb2\" (UID: \"53bd305a-c566-47e2-bbfe-cddf73635d77\") " pod="openshift-ingress-canary/ingress-canary-llvb2" Sep 09 10:26:35.383735 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.383552 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk26p\" (UniqueName: \"kubernetes.io/projected/53bd305a-c566-47e2-bbfe-cddf73635d77-kube-api-access-rk26p\") pod \"ingress-canary-llvb2\" (UID: \"53bd305a-c566-47e2-bbfe-cddf73635d77\") " pod="openshift-ingress-canary/ingress-canary-llvb2" Sep 09 10:26:35.384025 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.384004 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmqkd\" (UniqueName: \"kubernetes.io/projected/88316610-e58d-4f66-ae6c-eb81674b4f43-kube-api-access-hmqkd\") pod \"dns-default-jjw92\" (UID: \"88316610-e58d-4f66-ae6c-eb81674b4f43\") " pod="openshift-dns/dns-default-jjw92" Sep 09 10:26:35.460133 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.460045 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-llvb2" Sep 09 10:26:35.785260 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.785177 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-llvb2"] Sep 09 10:26:35.791337 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:26:35.791301 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53bd305a_c566_47e2_bbfe_cddf73635d77.slice/crio-bdb57ed6011da021a04d98ed346c150a2bf96ae3470c896cdf630c55329056de WatchSource:0}: Error finding container bdb57ed6011da021a04d98ed346c150a2bf96ae3470c896cdf630c55329056de: Status 404 returned error can't find the container with id bdb57ed6011da021a04d98ed346c150a2bf96ae3470c896cdf630c55329056de Sep 09 10:26:35.874991 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:35.874957 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/88316610-e58d-4f66-ae6c-eb81674b4f43-metrics-tls\") pod \"dns-default-jjw92\" (UID: \"88316610-e58d-4f66-ae6c-eb81674b4f43\") " pod="openshift-dns/dns-default-jjw92" Sep 09 10:26:35.875162 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:35.875114 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 10:26:35.875212 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:35.875184 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88316610-e58d-4f66-ae6c-eb81674b4f43-metrics-tls podName:88316610-e58d-4f66-ae6c-eb81674b4f43 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:36.875165741 +0000 UTC m=+37.049768043 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/88316610-e58d-4f66-ae6c-eb81674b4f43-metrics-tls") pod "dns-default-jjw92" (UID: "88316610-e58d-4f66-ae6c-eb81674b4f43") : secret "dns-default-metrics-tls" not found Sep 09 10:26:36.328873 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:36.328835 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:26:36.329061 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:36.328880 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:26:36.329061 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:36.328839 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:26:36.334005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:36.332046 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 10:26:36.334005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:36.332347 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 10:26:36.334005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:36.332598 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 10:26:36.334005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:36.332950 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hq5pc\"" Sep 09 10:26:36.334005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:36.333227 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ffrc8\"" Sep 09 10:26:36.334005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:36.333434 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 10:26:36.511266 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:36.511228 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-llvb2" event={"ID":"53bd305a-c566-47e2-bbfe-cddf73635d77","Type":"ContainerStarted","Data":"bdb57ed6011da021a04d98ed346c150a2bf96ae3470c896cdf630c55329056de"} Sep 09 10:26:36.883392 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:36.883298 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/88316610-e58d-4f66-ae6c-eb81674b4f43-metrics-tls\") pod \"dns-default-jjw92\" (UID: \"88316610-e58d-4f66-ae6c-eb81674b4f43\") " pod="openshift-dns/dns-default-jjw92" Sep 09 10:26:36.883588 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:36.883516 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 10:26:36.883653 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:36.883606 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88316610-e58d-4f66-ae6c-eb81674b4f43-metrics-tls podName:88316610-e58d-4f66-ae6c-eb81674b4f43 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:38.883582065 +0000 UTC m=+39.058184372 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/88316610-e58d-4f66-ae6c-eb81674b4f43-metrics-tls") pod "dns-default-jjw92" (UID: "88316610-e58d-4f66-ae6c-eb81674b4f43") : secret "dns-default-metrics-tls" not found Sep 09 10:26:38.900380 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:38.900337 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/88316610-e58d-4f66-ae6c-eb81674b4f43-metrics-tls\") pod \"dns-default-jjw92\" (UID: \"88316610-e58d-4f66-ae6c-eb81674b4f43\") " pod="openshift-dns/dns-default-jjw92" Sep 09 10:26:38.900952 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:38.900523 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 10:26:38.900952 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:38.900608 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88316610-e58d-4f66-ae6c-eb81674b4f43-metrics-tls podName:88316610-e58d-4f66-ae6c-eb81674b4f43 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:42.900586016 +0000 UTC m=+43.075188322 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/88316610-e58d-4f66-ae6c-eb81674b4f43-metrics-tls") pod "dns-default-jjw92" (UID: "88316610-e58d-4f66-ae6c-eb81674b4f43") : secret "dns-default-metrics-tls" not found Sep 09 10:26:39.524799 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:39.524763 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5dpbc" event={"ID":"f6aee692-b2c3-48d5-af2b-90bdeb0a29a7","Type":"ContainerStarted","Data":"5b234621318992bdec89edd1163acfb6acfc1bcee0b4d3bf96261d99bbc246ce"} Sep 09 10:26:39.526534 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:39.526480 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-llvb2" event={"ID":"53bd305a-c566-47e2-bbfe-cddf73635d77","Type":"ContainerStarted","Data":"05a5eb211951961b47d3d7592ad1ebb82d309bb30a1bb700db1cf644fec68b25"} Sep 09 10:26:39.569957 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:39.567841 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-llvb2" podStartSLOduration=1.028354468 podStartE2EDuration="4.567809525s" totalImagesPullingTime="3.539455057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:26:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:39.566957403 +0000 UTC m=+39.741559729" watchObservedRunningTime="2026-09-09 10:26:39.567809525 +0000 UTC m=+39.742411851" Sep 09 10:26:40.531751 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:40.531713 2559 generic.go:356] "Generic (PLEG): container finished" podID="f6aee692-b2c3-48d5-af2b-90bdeb0a29a7" containerID="5b234621318992bdec89edd1163acfb6acfc1bcee0b4d3bf96261d99bbc246ce" exitCode=0 Sep 09 10:26:40.532193 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:40.531805 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5dpbc" event={"ID":"f6aee692-b2c3-48d5-af2b-90bdeb0a29a7","Type":"ContainerDied","Data":"5b234621318992bdec89edd1163acfb6acfc1bcee0b4d3bf96261d99bbc246ce"} Sep 09 10:26:41.537031 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:41.536995 2559 generic.go:356] "Generic (PLEG): container finished" podID="f6aee692-b2c3-48d5-af2b-90bdeb0a29a7" containerID="f32be9e279012d644eec9e9a0b5d44d5660fe284d8a0677317af45a4cb124137" exitCode=0 Sep 09 10:26:41.537399 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:41.537052 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5dpbc" event={"ID":"f6aee692-b2c3-48d5-af2b-90bdeb0a29a7","Type":"ContainerDied","Data":"f32be9e279012d644eec9e9a0b5d44d5660fe284d8a0677317af45a4cb124137"} Sep 09 10:26:42.543277 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:42.543241 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5dpbc" event={"ID":"f6aee692-b2c3-48d5-af2b-90bdeb0a29a7","Type":"ContainerStarted","Data":"b9a66c7d0c73748f94c270ab1d18c76fae7a3b3495fc5a6548c5ecec5657014d"} Sep 09 10:26:42.572582 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:42.572420 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5dpbc" podStartSLOduration=9.39908058 podStartE2EDuration="42.572400713s" totalImagesPullingTime="33.173320133s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:26:00 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:26:42.571098982 +0000 UTC m=+42.745701322" watchObservedRunningTime="2026-09-09 10:26:42.572400713 +0000 UTC m=+42.747003038" Sep 09 10:26:42.928346 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:42.928306 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/88316610-e58d-4f66-ae6c-eb81674b4f43-metrics-tls\") pod \"dns-default-jjw92\" (UID: \"88316610-e58d-4f66-ae6c-eb81674b4f43\") " pod="openshift-dns/dns-default-jjw92" Sep 09 10:26:42.928520 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:42.928435 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 10:26:42.928575 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:42.928495 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88316610-e58d-4f66-ae6c-eb81674b4f43-metrics-tls podName:88316610-e58d-4f66-ae6c-eb81674b4f43 nodeName:}" failed. No retries permitted until 2026-09-09 10:26:50.928481444 +0000 UTC m=+51.103083747 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/88316610-e58d-4f66-ae6c-eb81674b4f43-metrics-tls") pod "dns-default-jjw92" (UID: "88316610-e58d-4f66-ae6c-eb81674b4f43") : secret "dns-default-metrics-tls" not found Sep 09 10:26:50.978967 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:26:50.978921 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/88316610-e58d-4f66-ae6c-eb81674b4f43-metrics-tls\") pod \"dns-default-jjw92\" (UID: \"88316610-e58d-4f66-ae6c-eb81674b4f43\") " pod="openshift-dns/dns-default-jjw92" Sep 09 10:26:50.979353 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:50.979075 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 10:26:50.979353 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:26:50.979139 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88316610-e58d-4f66-ae6c-eb81674b4f43-metrics-tls podName:88316610-e58d-4f66-ae6c-eb81674b4f43 nodeName:}" failed. No retries permitted until 2026-09-09 10:27:06.979121617 +0000 UTC m=+67.153723937 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/88316610-e58d-4f66-ae6c-eb81674b4f43-metrics-tls") pod "dns-default-jjw92" (UID: "88316610-e58d-4f66-ae6c-eb81674b4f43") : secret "dns-default-metrics-tls" not found Sep 09 10:27:00.506837 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:27:00.506807 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d22ff" Sep 09 10:27:05.070712 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:27:05.070649 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b697c056-99a2-4c91-a7c3-1f787553973e-original-pull-secret\") pod \"global-pull-secret-syncer-k9fqd\" (UID: \"b697c056-99a2-4c91-a7c3-1f787553973e\") " pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:27:05.070712 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:27:05.070718 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs\") pod \"network-metrics-daemon-c4s48\" (UID: \"1b8cf871-4dea-4ac5-868a-35b02f455e65\") " pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:27:05.073807 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:27:05.073780 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 10:27:05.073936 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:27:05.073873 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 10:27:05.080904 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:27:05.080887 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 10:27:05.080965 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:27:05.080952 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs podName:1b8cf871-4dea-4ac5-868a-35b02f455e65 nodeName:}" failed. No retries permitted until 2026-09-09 10:28:09.080934528 +0000 UTC m=+129.255536835 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs") pod "network-metrics-daemon-c4s48" (UID: "1b8cf871-4dea-4ac5-868a-35b02f455e65") : secret "metrics-daemon-secret" not found Sep 09 10:27:05.083415 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:27:05.083398 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b697c056-99a2-4c91-a7c3-1f787553973e-original-pull-secret\") pod \"global-pull-secret-syncer-k9fqd\" (UID: \"b697c056-99a2-4c91-a7c3-1f787553973e\") " pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:27:05.141759 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:27:05.141725 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9fqd" Sep 09 10:27:05.171491 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:27:05.171462 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54scz\" (UniqueName: \"kubernetes.io/projected/946804f0-f29b-4f4c-8990-3cc0bf40c79e-kube-api-access-54scz\") pod \"network-check-target-vsngp\" (UID: \"946804f0-f29b-4f4c-8990-3cc0bf40c79e\") " pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:27:05.174745 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:27:05.174721 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 10:27:05.185893 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:27:05.185871 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 10:27:05.195174 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:27:05.195144 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54scz\" (UniqueName: \"kubernetes.io/projected/946804f0-f29b-4f4c-8990-3cc0bf40c79e-kube-api-access-54scz\") pod \"network-check-target-vsngp\" (UID: \"946804f0-f29b-4f4c-8990-3cc0bf40c79e\") " pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:27:05.427724 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:27:05.427692 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-k9fqd"] Sep 09 10:27:05.429028 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:27:05.428995 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb697c056_99a2_4c91_a7c3_1f787553973e.slice/crio-49ac42d06dba86546d32b09ef584c6f40930d1f47e62c7819519fc344b73b891 WatchSource:0}: Error finding container 49ac42d06dba86546d32b09ef584c6f40930d1f47e62c7819519fc344b73b891: Status 404 returned error can't find the container with id 49ac42d06dba86546d32b09ef584c6f40930d1f47e62c7819519fc344b73b891 Sep 09 10:27:05.457429 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:27:05.457398 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ffrc8\"" Sep 09 10:27:05.465201 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:27:05.465179 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:27:05.599044 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:27:05.599006 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-k9fqd" event={"ID":"b697c056-99a2-4c91-a7c3-1f787553973e","Type":"ContainerStarted","Data":"49ac42d06dba86546d32b09ef584c6f40930d1f47e62c7819519fc344b73b891"} Sep 09 10:27:05.736856 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:27:05.736783 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vsngp"] Sep 09 10:27:05.739552 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:27:05.739528 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946804f0_f29b_4f4c_8990_3cc0bf40c79e.slice/crio-6bbbab06bd923618710386b4a5867f8c42db1c1bb577c813773e162257f08f7b WatchSource:0}: Error finding container 6bbbab06bd923618710386b4a5867f8c42db1c1bb577c813773e162257f08f7b: Status 404 returned error can't find the container with id 6bbbab06bd923618710386b4a5867f8c42db1c1bb577c813773e162257f08f7b Sep 09 10:27:06.603859 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:27:06.603822 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vsngp" event={"ID":"946804f0-f29b-4f4c-8990-3cc0bf40c79e","Type":"ContainerStarted","Data":"6bbbab06bd923618710386b4a5867f8c42db1c1bb577c813773e162257f08f7b"} Sep 09 10:27:06.983248 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:27:06.983154 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/88316610-e58d-4f66-ae6c-eb81674b4f43-metrics-tls\") pod \"dns-default-jjw92\" (UID: \"88316610-e58d-4f66-ae6c-eb81674b4f43\") " pod="openshift-dns/dns-default-jjw92" Sep 09 10:27:06.983414 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:27:06.983312 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 10:27:06.983414 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:27:06.983381 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88316610-e58d-4f66-ae6c-eb81674b4f43-metrics-tls podName:88316610-e58d-4f66-ae6c-eb81674b4f43 nodeName:}" failed. No retries permitted until 2026-09-09 10:27:38.983361287 +0000 UTC m=+99.157963607 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/88316610-e58d-4f66-ae6c-eb81674b4f43-metrics-tls") pod "dns-default-jjw92" (UID: "88316610-e58d-4f66-ae6c-eb81674b4f43") : secret "dns-default-metrics-tls" not found Sep 09 10:27:10.617001 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:27:10.616962 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vsngp" event={"ID":"946804f0-f29b-4f4c-8990-3cc0bf40c79e","Type":"ContainerStarted","Data":"1269c38191a7ce65226121bbc1b392d84cb71298f44b0dc0da6398d56fe391f4"} Sep 09 10:27:10.617462 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:27:10.617056 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:27:10.618382 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:27:10.618355 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-k9fqd" event={"ID":"b697c056-99a2-4c91-a7c3-1f787553973e","Type":"ContainerStarted","Data":"7d28835fa1e27f306e924d88638eb3c783d8836acc77b54e8c70b2d400ebd030"} Sep 09 10:27:10.635946 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:27:10.635884 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vsngp" podStartSLOduration=66.184465982 podStartE2EDuration="1m10.635869267s" totalImagesPullingTime="4.451403285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:27:10.63521486 +0000 UTC m=+70.809817186" watchObservedRunningTime="2026-09-09 10:27:10.635869267 +0000 UTC m=+70.810471586" Sep 09 10:27:10.654370 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:27:10.654315 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-k9fqd" podStartSLOduration=65.891978613 podStartE2EDuration="1m10.654299726s" totalImagesPullingTime="4.762321113s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:27:10.652918756 +0000 UTC m=+70.827521081" watchObservedRunningTime="2026-09-09 10:27:10.654299726 +0000 UTC m=+70.828902082" Sep 09 10:27:39.082078 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:27:39.082036 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/88316610-e58d-4f66-ae6c-eb81674b4f43-metrics-tls\") pod \"dns-default-jjw92\" (UID: \"88316610-e58d-4f66-ae6c-eb81674b4f43\") " pod="openshift-dns/dns-default-jjw92" Sep 09 10:27:39.082543 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:27:39.082170 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 10:27:39.082543 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:27:39.082248 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88316610-e58d-4f66-ae6c-eb81674b4f43-metrics-tls podName:88316610-e58d-4f66-ae6c-eb81674b4f43 nodeName:}" failed. No retries permitted until 2026-09-09 10:28:43.082232256 +0000 UTC m=+163.256834558 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/88316610-e58d-4f66-ae6c-eb81674b4f43-metrics-tls") pod "dns-default-jjw92" (UID: "88316610-e58d-4f66-ae6c-eb81674b4f43") : secret "dns-default-metrics-tls" not found Sep 09 10:27:41.623770 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:27:41.623740 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vsngp" Sep 09 10:28:01.195665 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.195625 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c8f4c5bd-hqchf"] Sep 09 10:28:01.197403 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.197387 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c8f4c5bd-hqchf" Sep 09 10:28:01.201164 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.201135 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-c4dn7\"" Sep 09 10:28:01.201164 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.201152 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 09 10:28:01.201382 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.201178 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 09 10:28:01.201382 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.201199 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 09 10:28:01.201382 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.201152 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 09 10:28:01.209163 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.209136 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c8f4c5bd-hqchf"] Sep 09 10:28:01.212432 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.212410 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h"] Sep 09 10:28:01.214137 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.214120 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" Sep 09 10:28:01.216538 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.216514 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 09 10:28:01.216728 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.216705 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 09 10:28:01.216817 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.216726 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 09 10:28:01.216817 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.216804 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 09 10:28:01.225692 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.225667 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f178bb48-0f14-4037-85d0-2790556c4048-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-64c8f4c5bd-hqchf\" (UID: \"f178bb48-0f14-4037-85d0-2790556c4048\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c8f4c5bd-hqchf" Sep 09 10:28:01.225789 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.225700 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnf96\" (UniqueName: \"kubernetes.io/projected/f178bb48-0f14-4037-85d0-2790556c4048-kube-api-access-wnf96\") pod \"managed-serviceaccount-addon-agent-64c8f4c5bd-hqchf\" (UID: \"f178bb48-0f14-4037-85d0-2790556c4048\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c8f4c5bd-hqchf" Sep 09 10:28:01.227226 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.227206 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h"] Sep 09 10:28:01.326243 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.326211 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f178bb48-0f14-4037-85d0-2790556c4048-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-64c8f4c5bd-hqchf\" (UID: \"f178bb48-0f14-4037-85d0-2790556c4048\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c8f4c5bd-hqchf" Sep 09 10:28:01.326243 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.326248 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnf96\" (UniqueName: \"kubernetes.io/projected/f178bb48-0f14-4037-85d0-2790556c4048-kube-api-access-wnf96\") pod \"managed-serviceaccount-addon-agent-64c8f4c5bd-hqchf\" (UID: \"f178bb48-0f14-4037-85d0-2790556c4048\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c8f4c5bd-hqchf" Sep 09 10:28:01.326531 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.326278 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5f9d66b65f-xn58h\" (UID: \"3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" Sep 09 10:28:01.326531 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.326304 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5f9d66b65f-xn58h\" (UID: \"3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" Sep 09 10:28:01.326531 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.326331 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a-ca\") pod \"cluster-proxy-proxy-agent-5f9d66b65f-xn58h\" (UID: \"3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" Sep 09 10:28:01.326531 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.326376 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a-hub\") pod \"cluster-proxy-proxy-agent-5f9d66b65f-xn58h\" (UID: \"3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" Sep 09 10:28:01.326703 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.326518 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rhp4\" (UniqueName: \"kubernetes.io/projected/3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a-kube-api-access-7rhp4\") pod \"cluster-proxy-proxy-agent-5f9d66b65f-xn58h\" (UID: \"3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" Sep 09 10:28:01.326703 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.326574 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5f9d66b65f-xn58h\" (UID: \"3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" Sep 09 10:28:01.328821 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.328800 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f178bb48-0f14-4037-85d0-2790556c4048-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-64c8f4c5bd-hqchf\" (UID: \"f178bb48-0f14-4037-85d0-2790556c4048\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c8f4c5bd-hqchf" Sep 09 10:28:01.335831 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.335802 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnf96\" (UniqueName: \"kubernetes.io/projected/f178bb48-0f14-4037-85d0-2790556c4048-kube-api-access-wnf96\") pod \"managed-serviceaccount-addon-agent-64c8f4c5bd-hqchf\" (UID: \"f178bb48-0f14-4037-85d0-2790556c4048\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c8f4c5bd-hqchf" Sep 09 10:28:01.427433 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.427389 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a-ca\") pod \"cluster-proxy-proxy-agent-5f9d66b65f-xn58h\" (UID: \"3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" Sep 09 10:28:01.427660 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.427446 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a-hub\") pod \"cluster-proxy-proxy-agent-5f9d66b65f-xn58h\" (UID: \"3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" Sep 09 10:28:01.427660 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.427486 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rhp4\" (UniqueName: \"kubernetes.io/projected/3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a-kube-api-access-7rhp4\") pod \"cluster-proxy-proxy-agent-5f9d66b65f-xn58h\" (UID: \"3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" Sep 09 10:28:01.427780 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.427651 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5f9d66b65f-xn58h\" (UID: \"3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" Sep 09 10:28:01.427780 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.427705 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5f9d66b65f-xn58h\" (UID: \"3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" Sep 09 10:28:01.427780 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.427735 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5f9d66b65f-xn58h\" (UID: \"3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" Sep 09 10:28:01.428475 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.428443 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5f9d66b65f-xn58h\" (UID: \"3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" Sep 09 10:28:01.429978 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.429954 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a-hub\") pod \"cluster-proxy-proxy-agent-5f9d66b65f-xn58h\" (UID: \"3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" Sep 09 10:28:01.430062 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.429979 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5f9d66b65f-xn58h\" (UID: \"3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" Sep 09 10:28:01.430062 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.429993 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a-ca\") pod \"cluster-proxy-proxy-agent-5f9d66b65f-xn58h\" (UID: \"3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" Sep 09 10:28:01.430206 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.430190 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5f9d66b65f-xn58h\" (UID: \"3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" Sep 09 10:28:01.436867 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.436845 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rhp4\" (UniqueName: \"kubernetes.io/projected/3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a-kube-api-access-7rhp4\") pod \"cluster-proxy-proxy-agent-5f9d66b65f-xn58h\" (UID: \"3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" Sep 09 10:28:01.516982 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.516888 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c8f4c5bd-hqchf" Sep 09 10:28:01.523708 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.523577 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" Sep 09 10:28:01.784601 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.784518 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c8f4c5bd-hqchf"] Sep 09 10:28:01.788528 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:28:01.788482 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf178bb48_0f14_4037_85d0_2790556c4048.slice/crio-6395b061d152b2e36e78430403e2696fab6e121e9fa8b025603c1f2366fbe9db WatchSource:0}: Error finding container 6395b061d152b2e36e78430403e2696fab6e121e9fa8b025603c1f2366fbe9db: Status 404 returned error can't find the container with id 6395b061d152b2e36e78430403e2696fab6e121e9fa8b025603c1f2366fbe9db Sep 09 10:28:01.806864 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:01.806829 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h"] Sep 09 10:28:01.810003 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:28:01.809978 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3384b0aa_c7d3_4e1c_b73f_6b96a924ed7a.slice/crio-d8d7baa4382058192597f255bb81709020fb20f185ff78f9c18ece25f6a22957 WatchSource:0}: Error finding container d8d7baa4382058192597f255bb81709020fb20f185ff78f9c18ece25f6a22957: Status 404 returned error can't find the container with id d8d7baa4382058192597f255bb81709020fb20f185ff78f9c18ece25f6a22957 Sep 09 10:28:02.766692 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:02.766610 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c8f4c5bd-hqchf" event={"ID":"f178bb48-0f14-4037-85d0-2790556c4048","Type":"ContainerStarted","Data":"6395b061d152b2e36e78430403e2696fab6e121e9fa8b025603c1f2366fbe9db"} Sep 09 10:28:02.769165 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:02.769102 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" event={"ID":"3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a","Type":"ContainerStarted","Data":"d8d7baa4382058192597f255bb81709020fb20f185ff78f9c18ece25f6a22957"} Sep 09 10:28:05.782949 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:05.782915 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c8f4c5bd-hqchf" event={"ID":"f178bb48-0f14-4037-85d0-2790556c4048","Type":"ContainerStarted","Data":"de0ab40ecb4a3ad6d3096ced35109c438b331016a1003e2c919be876242b4769"} Sep 09 10:28:05.784553 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:05.784491 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" event={"ID":"3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a","Type":"ContainerStarted","Data":"85a33aa3eff274963d29e6fb8763375c5a57f4a34f582544501b3a1f80759f1c"} Sep 09 10:28:05.799472 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:05.799422 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c8f4c5bd-hqchf" podStartSLOduration=0.901975848 podStartE2EDuration="4.799406999s" totalImagesPullingTime="3.897431151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:28:05.799007849 +0000 UTC m=+125.973610174" watchObservedRunningTime="2026-09-09 10:28:05.799406999 +0000 UTC m=+125.974009324" Sep 09 10:28:06.790443 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:06.790404 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" event={"ID":"3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a","Type":"ContainerStarted","Data":"ebfe7883c08a5c081d125ad8cc10eba419db4768e01028a964035b21d86efa7f"} Sep 09 10:28:06.790443 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:06.790450 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" event={"ID":"3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a","Type":"ContainerStarted","Data":"601a0bb49653611ad75da3c44873c8e18007d8d1491a0f90a084f67091441eb9"} Sep 09 10:28:06.810596 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:06.810541 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" podStartSLOduration=1.915027139 podStartE2EDuration="5.810517471s" totalImagesPullingTime="3.895490332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:28:06.810419139 +0000 UTC m=+126.985021475" watchObservedRunningTime="2026-09-09 10:28:06.810517471 +0000 UTC m=+126.985119787" Sep 09 10:28:09.082338 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:09.082291 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs\") pod \"network-metrics-daemon-c4s48\" (UID: \"1b8cf871-4dea-4ac5-868a-35b02f455e65\") " pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:28:09.082770 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:28:09.082457 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 10:28:09.082770 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:28:09.082550 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs podName:1b8cf871-4dea-4ac5-868a-35b02f455e65 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:11.082533409 +0000 UTC m=+251.257135712 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs") pod "network-metrics-daemon-c4s48" (UID: "1b8cf871-4dea-4ac5-868a-35b02f455e65") : secret "metrics-daemon-secret" not found Sep 09 10:28:38.168471 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:28:38.168428 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-jjw92" podUID="88316610-e58d-4f66-ae6c-eb81674b4f43" Sep 09 10:28:38.886150 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:38.886118 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jjw92" Sep 09 10:28:39.348548 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:28:39.348484 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-c4s48" podUID="1b8cf871-4dea-4ac5-868a-35b02f455e65" Sep 09 10:28:43.121954 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:43.121906 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/88316610-e58d-4f66-ae6c-eb81674b4f43-metrics-tls\") pod \"dns-default-jjw92\" (UID: \"88316610-e58d-4f66-ae6c-eb81674b4f43\") " pod="openshift-dns/dns-default-jjw92" Sep 09 10:28:43.122354 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:28:43.122048 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 10:28:43.122354 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:28:43.122118 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88316610-e58d-4f66-ae6c-eb81674b4f43-metrics-tls podName:88316610-e58d-4f66-ae6c-eb81674b4f43 nodeName:}" failed. No retries permitted until 2026-09-09 10:30:45.122101448 +0000 UTC m=+285.296703757 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/88316610-e58d-4f66-ae6c-eb81674b4f43-metrics-tls") pod "dns-default-jjw92" (UID: "88316610-e58d-4f66-ae6c-eb81674b4f43") : secret "dns-default-metrics-tls" not found Sep 09 10:28:53.328510 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:28:53.328473 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:30:06.142050 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:06.142015 2559 generic.go:356] "Generic (PLEG): container finished" podID="f178bb48-0f14-4037-85d0-2790556c4048" containerID="de0ab40ecb4a3ad6d3096ced35109c438b331016a1003e2c919be876242b4769" exitCode=255 Sep 09 10:30:06.142488 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:06.142058 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c8f4c5bd-hqchf" event={"ID":"f178bb48-0f14-4037-85d0-2790556c4048","Type":"ContainerDied","Data":"de0ab40ecb4a3ad6d3096ced35109c438b331016a1003e2c919be876242b4769"} Sep 09 10:30:06.142488 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:06.142334 2559 scope.go:117] "RemoveContainer" containerID="de0ab40ecb4a3ad6d3096ced35109c438b331016a1003e2c919be876242b4769" Sep 09 10:30:07.147095 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:07.147057 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c8f4c5bd-hqchf" event={"ID":"f178bb48-0f14-4037-85d0-2790556c4048","Type":"ContainerStarted","Data":"5c57005ed240755a27ed31f18512afba234f37ea75f113c9d9f5775c1f1e90e5"} Sep 09 10:30:11.141606 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:11.141546 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs\") pod \"network-metrics-daemon-c4s48\" (UID: \"1b8cf871-4dea-4ac5-868a-35b02f455e65\") " pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:30:11.142034 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:30:11.141705 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 10:30:11.142034 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:30:11.141781 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs podName:1b8cf871-4dea-4ac5-868a-35b02f455e65 nodeName:}" failed. No retries permitted until 2026-09-09 10:32:13.141763685 +0000 UTC m=+373.316365988 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs") pod "network-metrics-daemon-c4s48" (UID: "1b8cf871-4dea-4ac5-868a-35b02f455e65") : secret "metrics-daemon-secret" not found Sep 09 10:30:11.492772 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:11.492681 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rgrgx"] Sep 09 10:30:11.495032 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:11.495005 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rgrgx" Sep 09 10:30:11.497820 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:11.497799 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 09 10:30:11.497956 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:11.497816 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 09 10:30:11.498855 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:11.498825 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 09 10:30:11.498968 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:11.498853 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-89f9s\"" Sep 09 10:30:11.498968 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:11.498831 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 09 10:30:11.508882 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:11.508856 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rgrgx"] Sep 09 10:30:11.645648 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:11.645603 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44bad09f-4826-4024-8221-7abbd2051cba-crio-socket\") pod \"insights-runtime-extractor-rgrgx\" (UID: \"44bad09f-4826-4024-8221-7abbd2051cba\") " pod="openshift-insights/insights-runtime-extractor-rgrgx" Sep 09 10:30:11.645648 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:11.645651 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44bad09f-4826-4024-8221-7abbd2051cba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rgrgx\" (UID: \"44bad09f-4826-4024-8221-7abbd2051cba\") " pod="openshift-insights/insights-runtime-extractor-rgrgx" Sep 09 10:30:11.645889 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:11.645680 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44bad09f-4826-4024-8221-7abbd2051cba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rgrgx\" (UID: \"44bad09f-4826-4024-8221-7abbd2051cba\") " pod="openshift-insights/insights-runtime-extractor-rgrgx" Sep 09 10:30:11.645889 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:11.645729 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlk87\" (UniqueName: \"kubernetes.io/projected/44bad09f-4826-4024-8221-7abbd2051cba-kube-api-access-hlk87\") pod \"insights-runtime-extractor-rgrgx\" (UID: \"44bad09f-4826-4024-8221-7abbd2051cba\") " pod="openshift-insights/insights-runtime-extractor-rgrgx" Sep 09 10:30:11.645889 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:11.645769 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/44bad09f-4826-4024-8221-7abbd2051cba-data-volume\") pod \"insights-runtime-extractor-rgrgx\" (UID: \"44bad09f-4826-4024-8221-7abbd2051cba\") " pod="openshift-insights/insights-runtime-extractor-rgrgx" Sep 09 10:30:11.746701 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:11.746604 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44bad09f-4826-4024-8221-7abbd2051cba-crio-socket\") pod \"insights-runtime-extractor-rgrgx\" (UID: \"44bad09f-4826-4024-8221-7abbd2051cba\") " pod="openshift-insights/insights-runtime-extractor-rgrgx" Sep 09 10:30:11.746701 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:11.746650 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44bad09f-4826-4024-8221-7abbd2051cba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rgrgx\" (UID: \"44bad09f-4826-4024-8221-7abbd2051cba\") " pod="openshift-insights/insights-runtime-extractor-rgrgx" Sep 09 10:30:11.746701 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:11.746667 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44bad09f-4826-4024-8221-7abbd2051cba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rgrgx\" (UID: \"44bad09f-4826-4024-8221-7abbd2051cba\") " pod="openshift-insights/insights-runtime-extractor-rgrgx" Sep 09 10:30:11.746701 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:11.746683 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlk87\" (UniqueName: \"kubernetes.io/projected/44bad09f-4826-4024-8221-7abbd2051cba-kube-api-access-hlk87\") pod \"insights-runtime-extractor-rgrgx\" (UID: \"44bad09f-4826-4024-8221-7abbd2051cba\") " pod="openshift-insights/insights-runtime-extractor-rgrgx" Sep 09 10:30:11.746701 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:11.746701 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/44bad09f-4826-4024-8221-7abbd2051cba-data-volume\") pod \"insights-runtime-extractor-rgrgx\" (UID: \"44bad09f-4826-4024-8221-7abbd2051cba\") " pod="openshift-insights/insights-runtime-extractor-rgrgx" Sep 09 10:30:11.747043 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:11.746757 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44bad09f-4826-4024-8221-7abbd2051cba-crio-socket\") pod \"insights-runtime-extractor-rgrgx\" (UID: \"44bad09f-4826-4024-8221-7abbd2051cba\") " pod="openshift-insights/insights-runtime-extractor-rgrgx" Sep 09 10:30:11.747043 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:30:11.746767 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 10:30:11.747043 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:30:11.746797 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 10:30:11.747043 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:30:11.746867 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44bad09f-4826-4024-8221-7abbd2051cba-kube-rbac-proxy-cm podName:44bad09f-4826-4024-8221-7abbd2051cba nodeName:}" failed. No retries permitted until 2026-09-09 10:30:12.246847291 +0000 UTC m=+252.421449605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44bad09f-4826-4024-8221-7abbd2051cba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rgrgx" (UID: "44bad09f-4826-4024-8221-7abbd2051cba") : configmap "kube-rbac-proxy" not found Sep 09 10:30:11.747043 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:30:11.746965 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44bad09f-4826-4024-8221-7abbd2051cba-insights-runtime-extractor-tls podName:44bad09f-4826-4024-8221-7abbd2051cba nodeName:}" failed. No retries permitted until 2026-09-09 10:30:12.246941601 +0000 UTC m=+252.421543910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44bad09f-4826-4024-8221-7abbd2051cba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rgrgx" (UID: "44bad09f-4826-4024-8221-7abbd2051cba") : secret "insights-runtime-extractor-tls" not found Sep 09 10:30:11.747214 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:11.747085 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/44bad09f-4826-4024-8221-7abbd2051cba-data-volume\") pod \"insights-runtime-extractor-rgrgx\" (UID: \"44bad09f-4826-4024-8221-7abbd2051cba\") " pod="openshift-insights/insights-runtime-extractor-rgrgx" Sep 09 10:30:11.759663 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:11.759630 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlk87\" (UniqueName: \"kubernetes.io/projected/44bad09f-4826-4024-8221-7abbd2051cba-kube-api-access-hlk87\") pod \"insights-runtime-extractor-rgrgx\" (UID: \"44bad09f-4826-4024-8221-7abbd2051cba\") " pod="openshift-insights/insights-runtime-extractor-rgrgx" Sep 09 10:30:12.251004 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:12.250964 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44bad09f-4826-4024-8221-7abbd2051cba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rgrgx\" (UID: \"44bad09f-4826-4024-8221-7abbd2051cba\") " pod="openshift-insights/insights-runtime-extractor-rgrgx" Sep 09 10:30:12.251004 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:12.251007 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44bad09f-4826-4024-8221-7abbd2051cba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rgrgx\" (UID: \"44bad09f-4826-4024-8221-7abbd2051cba\") " pod="openshift-insights/insights-runtime-extractor-rgrgx" Sep 09 10:30:12.251490 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:30:12.251089 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 10:30:12.251490 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:30:12.251159 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44bad09f-4826-4024-8221-7abbd2051cba-kube-rbac-proxy-cm podName:44bad09f-4826-4024-8221-7abbd2051cba nodeName:}" failed. No retries permitted until 2026-09-09 10:30:13.251143436 +0000 UTC m=+253.425745739 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44bad09f-4826-4024-8221-7abbd2051cba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rgrgx" (UID: "44bad09f-4826-4024-8221-7abbd2051cba") : configmap "kube-rbac-proxy" not found Sep 09 10:30:12.251490 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:30:12.251181 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 10:30:12.251490 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:30:12.251233 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44bad09f-4826-4024-8221-7abbd2051cba-insights-runtime-extractor-tls podName:44bad09f-4826-4024-8221-7abbd2051cba nodeName:}" failed. No retries permitted until 2026-09-09 10:30:13.251218716 +0000 UTC m=+253.425821019 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44bad09f-4826-4024-8221-7abbd2051cba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rgrgx" (UID: "44bad09f-4826-4024-8221-7abbd2051cba") : secret "insights-runtime-extractor-tls" not found Sep 09 10:30:13.238595 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:13.238568 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4lsnr_13c59e77-04fc-4549-8402-df18b180161f/dns-node-resolver/0.log" Sep 09 10:30:13.257847 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:13.257816 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44bad09f-4826-4024-8221-7abbd2051cba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rgrgx\" (UID: \"44bad09f-4826-4024-8221-7abbd2051cba\") " pod="openshift-insights/insights-runtime-extractor-rgrgx" Sep 09 10:30:13.258165 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:13.257853 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44bad09f-4826-4024-8221-7abbd2051cba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rgrgx\" (UID: \"44bad09f-4826-4024-8221-7abbd2051cba\") " pod="openshift-insights/insights-runtime-extractor-rgrgx" Sep 09 10:30:13.258165 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:30:13.257941 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 10:30:13.258165 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:30:13.257997 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 10:30:13.258165 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:30:13.258006 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44bad09f-4826-4024-8221-7abbd2051cba-kube-rbac-proxy-cm podName:44bad09f-4826-4024-8221-7abbd2051cba nodeName:}" failed. No retries permitted until 2026-09-09 10:30:15.257987555 +0000 UTC m=+255.432589878 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44bad09f-4826-4024-8221-7abbd2051cba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rgrgx" (UID: "44bad09f-4826-4024-8221-7abbd2051cba") : configmap "kube-rbac-proxy" not found Sep 09 10:30:13.258165 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:30:13.258032 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44bad09f-4826-4024-8221-7abbd2051cba-insights-runtime-extractor-tls podName:44bad09f-4826-4024-8221-7abbd2051cba nodeName:}" failed. No retries permitted until 2026-09-09 10:30:15.258022421 +0000 UTC m=+255.432624725 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44bad09f-4826-4024-8221-7abbd2051cba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rgrgx" (UID: "44bad09f-4826-4024-8221-7abbd2051cba") : secret "insights-runtime-extractor-tls" not found Sep 09 10:30:14.438413 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:14.438387 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bq9w9_784872fd-624e-4323-862a-f1a040222214/node-ca/0.log" Sep 09 10:30:15.239103 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:15.239067 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-llvb2_53bd305a-c566-47e2-bbfe-cddf73635d77/serve-healthcheck-canary/0.log" Sep 09 10:30:15.271291 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:15.271254 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44bad09f-4826-4024-8221-7abbd2051cba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rgrgx\" (UID: \"44bad09f-4826-4024-8221-7abbd2051cba\") " pod="openshift-insights/insights-runtime-extractor-rgrgx" Sep 09 10:30:15.271403 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:15.271296 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44bad09f-4826-4024-8221-7abbd2051cba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rgrgx\" (UID: \"44bad09f-4826-4024-8221-7abbd2051cba\") " pod="openshift-insights/insights-runtime-extractor-rgrgx" Sep 09 10:30:15.271403 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:30:15.271375 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 10:30:15.271403 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:30:15.271394 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 10:30:15.271522 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:30:15.271452 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44bad09f-4826-4024-8221-7abbd2051cba-kube-rbac-proxy-cm podName:44bad09f-4826-4024-8221-7abbd2051cba nodeName:}" failed. No retries permitted until 2026-09-09 10:30:19.271437712 +0000 UTC m=+259.446040015 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44bad09f-4826-4024-8221-7abbd2051cba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rgrgx" (UID: "44bad09f-4826-4024-8221-7abbd2051cba") : configmap "kube-rbac-proxy" not found Sep 09 10:30:15.271570 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:30:15.271540 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44bad09f-4826-4024-8221-7abbd2051cba-insights-runtime-extractor-tls podName:44bad09f-4826-4024-8221-7abbd2051cba nodeName:}" failed. No retries permitted until 2026-09-09 10:30:19.271517369 +0000 UTC m=+259.446119692 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44bad09f-4826-4024-8221-7abbd2051cba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rgrgx" (UID: "44bad09f-4826-4024-8221-7abbd2051cba") : secret "insights-runtime-extractor-tls" not found Sep 09 10:30:19.302230 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:19.302184 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44bad09f-4826-4024-8221-7abbd2051cba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rgrgx\" (UID: \"44bad09f-4826-4024-8221-7abbd2051cba\") " pod="openshift-insights/insights-runtime-extractor-rgrgx" Sep 09 10:30:19.302230 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:19.302228 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44bad09f-4826-4024-8221-7abbd2051cba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rgrgx\" (UID: \"44bad09f-4826-4024-8221-7abbd2051cba\") " pod="openshift-insights/insights-runtime-extractor-rgrgx" Sep 09 10:30:19.302668 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:30:19.302313 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 10:30:19.302668 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:30:19.302334 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 10:30:19.302668 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:30:19.302388 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44bad09f-4826-4024-8221-7abbd2051cba-kube-rbac-proxy-cm podName:44bad09f-4826-4024-8221-7abbd2051cba nodeName:}" failed. No retries permitted until 2026-09-09 10:30:27.302371779 +0000 UTC m=+267.476974096 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44bad09f-4826-4024-8221-7abbd2051cba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rgrgx" (UID: "44bad09f-4826-4024-8221-7abbd2051cba") : configmap "kube-rbac-proxy" not found Sep 09 10:30:19.302668 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:30:19.302403 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44bad09f-4826-4024-8221-7abbd2051cba-insights-runtime-extractor-tls podName:44bad09f-4826-4024-8221-7abbd2051cba nodeName:}" failed. No retries permitted until 2026-09-09 10:30:27.302397365 +0000 UTC m=+267.476999668 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44bad09f-4826-4024-8221-7abbd2051cba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rgrgx" (UID: "44bad09f-4826-4024-8221-7abbd2051cba") : secret "insights-runtime-extractor-tls" not found Sep 09 10:30:27.362995 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:27.362901 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44bad09f-4826-4024-8221-7abbd2051cba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rgrgx\" (UID: \"44bad09f-4826-4024-8221-7abbd2051cba\") " pod="openshift-insights/insights-runtime-extractor-rgrgx" Sep 09 10:30:27.362995 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:27.362947 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44bad09f-4826-4024-8221-7abbd2051cba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rgrgx\" (UID: \"44bad09f-4826-4024-8221-7abbd2051cba\") " pod="openshift-insights/insights-runtime-extractor-rgrgx" Sep 09 10:30:27.363422 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:27.363408 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44bad09f-4826-4024-8221-7abbd2051cba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rgrgx\" (UID: \"44bad09f-4826-4024-8221-7abbd2051cba\") " pod="openshift-insights/insights-runtime-extractor-rgrgx" Sep 09 10:30:27.365250 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:27.365226 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44bad09f-4826-4024-8221-7abbd2051cba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rgrgx\" (UID: \"44bad09f-4826-4024-8221-7abbd2051cba\") " pod="openshift-insights/insights-runtime-extractor-rgrgx" Sep 09 10:30:27.404318 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:27.404288 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rgrgx" Sep 09 10:30:27.674546 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:27.674447 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rgrgx"] Sep 09 10:30:27.677788 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:30:27.677762 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44bad09f_4826_4024_8221_7abbd2051cba.slice/crio-240c2ae2c9b0d29b09cf3c8c05380ffcac926982e250b30f144fde6bf8685f5c WatchSource:0}: Error finding container 240c2ae2c9b0d29b09cf3c8c05380ffcac926982e250b30f144fde6bf8685f5c: Status 404 returned error can't find the container with id 240c2ae2c9b0d29b09cf3c8c05380ffcac926982e250b30f144fde6bf8685f5c Sep 09 10:30:28.213831 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:28.213799 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rgrgx" event={"ID":"44bad09f-4826-4024-8221-7abbd2051cba","Type":"ContainerStarted","Data":"bf61fa5613bfc19ad03421257ec34a70a29201c808c693531669dc92dda04510"} Sep 09 10:30:28.214010 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:28.213857 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rgrgx" event={"ID":"44bad09f-4826-4024-8221-7abbd2051cba","Type":"ContainerStarted","Data":"240c2ae2c9b0d29b09cf3c8c05380ffcac926982e250b30f144fde6bf8685f5c"} Sep 09 10:30:29.219423 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:29.219383 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rgrgx" event={"ID":"44bad09f-4826-4024-8221-7abbd2051cba","Type":"ContainerStarted","Data":"29032b69ffc850de1426e15a0111845d7f2a01d6499a459a835002fb294567d7"} Sep 09 10:30:30.224291 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:30.224195 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rgrgx" event={"ID":"44bad09f-4826-4024-8221-7abbd2051cba","Type":"ContainerStarted","Data":"57f523563ece8f192099662d9b076b48ac41659c468c3a5ce8823eafda6b2150"} Sep 09 10:30:30.243979 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:30.243931 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rgrgx" podStartSLOduration=17.171632914 podStartE2EDuration="19.243913679s" totalImagesPullingTime="2.072280765s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:30:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:30.243425949 +0000 UTC m=+270.418028276" watchObservedRunningTime="2026-09-09 10:30:30.243913679 +0000 UTC m=+270.418516003" Sep 09 10:30:35.548081 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:35.548049 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8fs4c"] Sep 09 10:30:35.551002 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:35.550971 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8fs4c" Sep 09 10:30:35.557540 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:35.557518 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 09 10:30:35.557653 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:35.557609 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-8qv59\"" Sep 09 10:30:35.558476 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:35.558449 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-dhvgm"] Sep 09 10:30:35.561209 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:35.561193 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-dhvgm" Sep 09 10:30:35.563609 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:35.563589 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 09 10:30:35.564067 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:35.564051 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 09 10:30:35.564261 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:35.564249 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-9xjrb\"" Sep 09 10:30:35.569826 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:35.569807 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8fs4c"] Sep 09 10:30:35.578885 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:35.578867 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-dhvgm"] Sep 09 10:30:35.726301 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:35.726259 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6401dd74-94ae-484b-bf82-fcd8d4ff6898-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-8fs4c\" (UID: \"6401dd74-94ae-484b-bf82-fcd8d4ff6898\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8fs4c" Sep 09 10:30:35.726301 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:35.726298 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd6cx\" (UniqueName: \"kubernetes.io/projected/b19da99e-eafb-470c-ab9b-fe1e6800a92d-kube-api-access-fd6cx\") pod \"downloads-85d98c998f-dhvgm\" (UID: \"b19da99e-eafb-470c-ab9b-fe1e6800a92d\") " pod="openshift-console/downloads-85d98c998f-dhvgm" Sep 09 10:30:35.827188 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:35.827157 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6401dd74-94ae-484b-bf82-fcd8d4ff6898-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-8fs4c\" (UID: \"6401dd74-94ae-484b-bf82-fcd8d4ff6898\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8fs4c" Sep 09 10:30:35.827332 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:35.827195 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd6cx\" (UniqueName: \"kubernetes.io/projected/b19da99e-eafb-470c-ab9b-fe1e6800a92d-kube-api-access-fd6cx\") pod \"downloads-85d98c998f-dhvgm\" (UID: \"b19da99e-eafb-470c-ab9b-fe1e6800a92d\") " pod="openshift-console/downloads-85d98c998f-dhvgm" Sep 09 10:30:35.829623 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:35.829602 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6401dd74-94ae-484b-bf82-fcd8d4ff6898-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-8fs4c\" (UID: \"6401dd74-94ae-484b-bf82-fcd8d4ff6898\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8fs4c" Sep 09 10:30:35.837674 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:35.837653 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd6cx\" (UniqueName: \"kubernetes.io/projected/b19da99e-eafb-470c-ab9b-fe1e6800a92d-kube-api-access-fd6cx\") pod \"downloads-85d98c998f-dhvgm\" (UID: \"b19da99e-eafb-470c-ab9b-fe1e6800a92d\") " pod="openshift-console/downloads-85d98c998f-dhvgm" Sep 09 10:30:35.860057 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:35.860033 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8fs4c" Sep 09 10:30:35.869934 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:35.869909 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-dhvgm" Sep 09 10:30:36.129269 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:36.129191 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8fs4c"] Sep 09 10:30:36.133812 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:30:36.133783 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6401dd74_94ae_484b_bf82_fcd8d4ff6898.slice/crio-9eab7871cdef68a9bd765face072b954defcedb4b25090b6474bfd6804715a98 WatchSource:0}: Error finding container 9eab7871cdef68a9bd765face072b954defcedb4b25090b6474bfd6804715a98: Status 404 returned error can't find the container with id 9eab7871cdef68a9bd765face072b954defcedb4b25090b6474bfd6804715a98 Sep 09 10:30:36.153005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:36.152983 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-dhvgm"] Sep 09 10:30:36.155566 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:30:36.155541 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb19da99e_eafb_470c_ab9b_fe1e6800a92d.slice/crio-a90f1c9a19243fe9a0f9f3654d03ec46b1a25e06ac920a8f285511cbe60c33db WatchSource:0}: Error finding container a90f1c9a19243fe9a0f9f3654d03ec46b1a25e06ac920a8f285511cbe60c33db: Status 404 returned error can't find the container with id a90f1c9a19243fe9a0f9f3654d03ec46b1a25e06ac920a8f285511cbe60c33db Sep 09 10:30:36.244861 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:36.244818 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-dhvgm" event={"ID":"b19da99e-eafb-470c-ab9b-fe1e6800a92d","Type":"ContainerStarted","Data":"a90f1c9a19243fe9a0f9f3654d03ec46b1a25e06ac920a8f285511cbe60c33db"} Sep 09 10:30:36.246199 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:36.246174 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8fs4c" event={"ID":"6401dd74-94ae-484b-bf82-fcd8d4ff6898","Type":"ContainerStarted","Data":"9eab7871cdef68a9bd765face072b954defcedb4b25090b6474bfd6804715a98"} Sep 09 10:30:39.256691 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:39.256595 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8fs4c" event={"ID":"6401dd74-94ae-484b-bf82-fcd8d4ff6898","Type":"ContainerStarted","Data":"12056b9673990d7d04f4798bcf423f76b858c650039a3d67e85df3639a771548"} Sep 09 10:30:39.257134 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:39.256842 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8fs4c" Sep 09 10:30:39.262436 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:39.262410 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8fs4c" Sep 09 10:30:39.272696 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:39.272651 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8fs4c" podStartSLOduration=1.444823122 podStartE2EDuration="4.272636365s" totalImagesPullingTime="2.827813243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:39.271953296 +0000 UTC m=+279.446555623" watchObservedRunningTime="2026-09-09 10:30:39.272636365 +0000 UTC m=+279.447238689" Sep 09 10:30:41.887120 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:30:41.887067 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-jjw92" podUID="88316610-e58d-4f66-ae6c-eb81674b4f43" Sep 09 10:30:42.266571 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:42.266485 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jjw92" Sep 09 10:30:45.207038 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:45.207003 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/88316610-e58d-4f66-ae6c-eb81674b4f43-metrics-tls\") pod \"dns-default-jjw92\" (UID: \"88316610-e58d-4f66-ae6c-eb81674b4f43\") " pod="openshift-dns/dns-default-jjw92" Sep 09 10:30:45.209714 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:45.209685 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/88316610-e58d-4f66-ae6c-eb81674b4f43-metrics-tls\") pod \"dns-default-jjw92\" (UID: \"88316610-e58d-4f66-ae6c-eb81674b4f43\") " pod="openshift-dns/dns-default-jjw92" Sep 09 10:30:45.271797 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:45.271765 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-272r8\"" Sep 09 10:30:45.278002 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:45.277977 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jjw92" Sep 09 10:30:45.565294 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:45.565210 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jjw92"] Sep 09 10:30:47.071667 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.071628 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-4p88h"] Sep 09 10:30:47.076518 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.076476 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p88h" Sep 09 10:30:47.080132 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.080105 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 09 10:30:47.081892 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.081861 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-d5lq4\"" Sep 09 10:30:47.082023 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.081940 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 09 10:30:47.082023 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.081945 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 09 10:30:47.082140 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.082025 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 09 10:30:47.082768 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.082735 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 09 10:30:47.090704 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.090670 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-4p88h"] Sep 09 10:30:47.101041 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.101011 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8jhbg"] Sep 09 10:30:47.104494 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.104471 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-qnjb6"] Sep 09 10:30:47.104623 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.104597 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8jhbg" Sep 09 10:30:47.107048 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.107022 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 09 10:30:47.107338 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.107287 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 09 10:30:47.107694 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.107675 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 09 10:30:47.107865 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.107849 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-qnjb6" Sep 09 10:30:47.107965 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.107947 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-x8vlp\"" Sep 09 10:30:47.110385 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.110367 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 09 10:30:47.110679 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.110662 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 09 10:30:47.111029 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.110999 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 09 10:30:47.111188 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.111168 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-b5jnj\"" Sep 09 10:30:47.122514 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.122413 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5vx8\" (UniqueName: \"kubernetes.io/projected/2a60cc00-e367-4273-8317-e3e33e871abc-kube-api-access-l5vx8\") pod \"openshift-state-metrics-657fd6c54-4p88h\" (UID: \"2a60cc00-e367-4273-8317-e3e33e871abc\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p88h" Sep 09 10:30:47.122653 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.122533 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a60cc00-e367-4273-8317-e3e33e871abc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-4p88h\" (UID: \"2a60cc00-e367-4273-8317-e3e33e871abc\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p88h" Sep 09 10:30:47.122653 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.122589 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a60cc00-e367-4273-8317-e3e33e871abc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-4p88h\" (UID: \"2a60cc00-e367-4273-8317-e3e33e871abc\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p88h" Sep 09 10:30:47.122653 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.122615 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a60cc00-e367-4273-8317-e3e33e871abc-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-4p88h\" (UID: \"2a60cc00-e367-4273-8317-e3e33e871abc\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p88h" Sep 09 10:30:47.123477 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.123450 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-qnjb6"] Sep 09 10:30:47.223436 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.223395 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0d829bc-3ccb-4fd7-ab82-bab2b9128c88-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8jhbg\" (UID: \"e0d829bc-3ccb-4fd7-ab82-bab2b9128c88\") " pod="openshift-monitoring/node-exporter-8jhbg" Sep 09 10:30:47.223655 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.223446 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a60cc00-e367-4273-8317-e3e33e871abc-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-4p88h\" (UID: \"2a60cc00-e367-4273-8317-e3e33e871abc\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p88h" Sep 09 10:30:47.223655 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.223471 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0d829bc-3ccb-4fd7-ab82-bab2b9128c88-metrics-client-ca\") pod \"node-exporter-8jhbg\" (UID: \"e0d829bc-3ccb-4fd7-ab82-bab2b9128c88\") " pod="openshift-monitoring/node-exporter-8jhbg" Sep 09 10:30:47.223655 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.223493 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e0d829bc-3ccb-4fd7-ab82-bab2b9128c88-node-exporter-tls\") pod \"node-exporter-8jhbg\" (UID: \"e0d829bc-3ccb-4fd7-ab82-bab2b9128c88\") " pod="openshift-monitoring/node-exporter-8jhbg" Sep 09 10:30:47.223655 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.223568 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-qnjb6\" (UID: \"3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qnjb6" Sep 09 10:30:47.223655 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.223589 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e0d829bc-3ccb-4fd7-ab82-bab2b9128c88-node-exporter-textfile\") pod \"node-exporter-8jhbg\" (UID: \"e0d829bc-3ccb-4fd7-ab82-bab2b9128c88\") " pod="openshift-monitoring/node-exporter-8jhbg" Sep 09 10:30:47.223655 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.223611 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lj4l\" (UniqueName: \"kubernetes.io/projected/3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3-kube-api-access-2lj4l\") pod \"kube-state-metrics-74cb57df7-qnjb6\" (UID: \"3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qnjb6" Sep 09 10:30:47.223956 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.223656 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a60cc00-e367-4273-8317-e3e33e871abc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-4p88h\" (UID: \"2a60cc00-e367-4273-8317-e3e33e871abc\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p88h" Sep 09 10:30:47.223956 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.223681 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e0d829bc-3ccb-4fd7-ab82-bab2b9128c88-node-exporter-accelerators-collector-config\") pod \"node-exporter-8jhbg\" (UID: \"e0d829bc-3ccb-4fd7-ab82-bab2b9128c88\") " pod="openshift-monitoring/node-exporter-8jhbg" Sep 09 10:30:47.223956 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.223819 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e0d829bc-3ccb-4fd7-ab82-bab2b9128c88-root\") pod \"node-exporter-8jhbg\" (UID: \"e0d829bc-3ccb-4fd7-ab82-bab2b9128c88\") " pod="openshift-monitoring/node-exporter-8jhbg" Sep 09 10:30:47.223956 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.223861 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5vx8\" (UniqueName: \"kubernetes.io/projected/2a60cc00-e367-4273-8317-e3e33e871abc-kube-api-access-l5vx8\") pod \"openshift-state-metrics-657fd6c54-4p88h\" (UID: \"2a60cc00-e367-4273-8317-e3e33e871abc\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p88h" Sep 09 10:30:47.223956 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.223887 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-qnjb6\" (UID: \"3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qnjb6" Sep 09 10:30:47.223956 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.223921 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-qnjb6\" (UID: \"3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qnjb6" Sep 09 10:30:47.223956 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.223956 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-qnjb6\" (UID: \"3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qnjb6" Sep 09 10:30:47.224273 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.224023 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e0d829bc-3ccb-4fd7-ab82-bab2b9128c88-node-exporter-wtmp\") pod \"node-exporter-8jhbg\" (UID: \"e0d829bc-3ccb-4fd7-ab82-bab2b9128c88\") " pod="openshift-monitoring/node-exporter-8jhbg" Sep 09 10:30:47.224273 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.224053 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-qnjb6\" (UID: \"3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qnjb6" Sep 09 10:30:47.224273 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.224078 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw685\" (UniqueName: \"kubernetes.io/projected/e0d829bc-3ccb-4fd7-ab82-bab2b9128c88-kube-api-access-pw685\") pod \"node-exporter-8jhbg\" (UID: \"e0d829bc-3ccb-4fd7-ab82-bab2b9128c88\") " pod="openshift-monitoring/node-exporter-8jhbg" Sep 09 10:30:47.224273 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.224104 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a60cc00-e367-4273-8317-e3e33e871abc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-4p88h\" (UID: \"2a60cc00-e367-4273-8317-e3e33e871abc\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p88h" Sep 09 10:30:47.224273 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.224147 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0d829bc-3ccb-4fd7-ab82-bab2b9128c88-sys\") pod \"node-exporter-8jhbg\" (UID: \"e0d829bc-3ccb-4fd7-ab82-bab2b9128c88\") " pod="openshift-monitoring/node-exporter-8jhbg" Sep 09 10:30:47.224273 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:30:47.224202 2559 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 09 10:30:47.224573 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:30:47.224278 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a60cc00-e367-4273-8317-e3e33e871abc-openshift-state-metrics-tls podName:2a60cc00-e367-4273-8317-e3e33e871abc nodeName:}" failed. No retries permitted until 2026-09-09 10:30:47.724257864 +0000 UTC m=+287.898860171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/2a60cc00-e367-4273-8317-e3e33e871abc-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-4p88h" (UID: "2a60cc00-e367-4273-8317-e3e33e871abc") : secret "openshift-state-metrics-tls" not found Sep 09 10:30:47.224573 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.224290 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a60cc00-e367-4273-8317-e3e33e871abc-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-4p88h\" (UID: \"2a60cc00-e367-4273-8317-e3e33e871abc\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p88h" Sep 09 10:30:47.226567 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.226495 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a60cc00-e367-4273-8317-e3e33e871abc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-4p88h\" (UID: \"2a60cc00-e367-4273-8317-e3e33e871abc\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p88h" Sep 09 10:30:47.234106 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.234080 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5vx8\" (UniqueName: \"kubernetes.io/projected/2a60cc00-e367-4273-8317-e3e33e871abc-kube-api-access-l5vx8\") pod \"openshift-state-metrics-657fd6c54-4p88h\" (UID: \"2a60cc00-e367-4273-8317-e3e33e871abc\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p88h" Sep 09 10:30:47.325315 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.325280 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0d829bc-3ccb-4fd7-ab82-bab2b9128c88-metrics-client-ca\") pod \"node-exporter-8jhbg\" (UID: \"e0d829bc-3ccb-4fd7-ab82-bab2b9128c88\") " pod="openshift-monitoring/node-exporter-8jhbg" Sep 09 10:30:47.325489 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.325330 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e0d829bc-3ccb-4fd7-ab82-bab2b9128c88-node-exporter-tls\") pod \"node-exporter-8jhbg\" (UID: \"e0d829bc-3ccb-4fd7-ab82-bab2b9128c88\") " pod="openshift-monitoring/node-exporter-8jhbg" Sep 09 10:30:47.325489 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.325388 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-qnjb6\" (UID: \"3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qnjb6" Sep 09 10:30:47.325489 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.325406 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e0d829bc-3ccb-4fd7-ab82-bab2b9128c88-node-exporter-textfile\") pod \"node-exporter-8jhbg\" (UID: \"e0d829bc-3ccb-4fd7-ab82-bab2b9128c88\") " pod="openshift-monitoring/node-exporter-8jhbg" Sep 09 10:30:47.325489 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.325428 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lj4l\" (UniqueName: \"kubernetes.io/projected/3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3-kube-api-access-2lj4l\") pod \"kube-state-metrics-74cb57df7-qnjb6\" (UID: \"3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qnjb6" Sep 09 10:30:47.325489 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.325476 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e0d829bc-3ccb-4fd7-ab82-bab2b9128c88-node-exporter-accelerators-collector-config\") pod \"node-exporter-8jhbg\" (UID: \"e0d829bc-3ccb-4fd7-ab82-bab2b9128c88\") " pod="openshift-monitoring/node-exporter-8jhbg" Sep 09 10:30:47.326000 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.325968 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e0d829bc-3ccb-4fd7-ab82-bab2b9128c88-root\") pod \"node-exporter-8jhbg\" (UID: \"e0d829bc-3ccb-4fd7-ab82-bab2b9128c88\") " pod="openshift-monitoring/node-exporter-8jhbg" Sep 09 10:30:47.326105 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.326027 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-qnjb6\" (UID: \"3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qnjb6" Sep 09 10:30:47.326105 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.326032 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e0d829bc-3ccb-4fd7-ab82-bab2b9128c88-root\") pod \"node-exporter-8jhbg\" (UID: \"e0d829bc-3ccb-4fd7-ab82-bab2b9128c88\") " pod="openshift-monitoring/node-exporter-8jhbg" Sep 09 10:30:47.326105 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.325978 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e0d829bc-3ccb-4fd7-ab82-bab2b9128c88-node-exporter-textfile\") pod \"node-exporter-8jhbg\" (UID: \"e0d829bc-3ccb-4fd7-ab82-bab2b9128c88\") " pod="openshift-monitoring/node-exporter-8jhbg" Sep 09 10:30:47.326105 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.326081 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e0d829bc-3ccb-4fd7-ab82-bab2b9128c88-node-exporter-accelerators-collector-config\") pod \"node-exporter-8jhbg\" (UID: \"e0d829bc-3ccb-4fd7-ab82-bab2b9128c88\") " pod="openshift-monitoring/node-exporter-8jhbg" Sep 09 10:30:47.326105 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.326082 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-qnjb6\" (UID: \"3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qnjb6" Sep 09 10:30:47.326353 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.326130 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-qnjb6\" (UID: \"3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qnjb6" Sep 09 10:30:47.326353 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.326175 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e0d829bc-3ccb-4fd7-ab82-bab2b9128c88-node-exporter-wtmp\") pod \"node-exporter-8jhbg\" (UID: \"e0d829bc-3ccb-4fd7-ab82-bab2b9128c88\") " pod="openshift-monitoring/node-exporter-8jhbg" Sep 09 10:30:47.326353 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.326201 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-qnjb6\" (UID: \"3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qnjb6" Sep 09 10:30:47.326353 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.326227 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-qnjb6\" (UID: \"3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qnjb6" Sep 09 10:30:47.326353 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.326231 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw685\" (UniqueName: \"kubernetes.io/projected/e0d829bc-3ccb-4fd7-ab82-bab2b9128c88-kube-api-access-pw685\") pod \"node-exporter-8jhbg\" (UID: \"e0d829bc-3ccb-4fd7-ab82-bab2b9128c88\") " pod="openshift-monitoring/node-exporter-8jhbg" Sep 09 10:30:47.326353 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.326281 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0d829bc-3ccb-4fd7-ab82-bab2b9128c88-sys\") pod \"node-exporter-8jhbg\" (UID: \"e0d829bc-3ccb-4fd7-ab82-bab2b9128c88\") " pod="openshift-monitoring/node-exporter-8jhbg" Sep 09 10:30:47.326353 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.326309 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0d829bc-3ccb-4fd7-ab82-bab2b9128c88-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8jhbg\" (UID: \"e0d829bc-3ccb-4fd7-ab82-bab2b9128c88\") " pod="openshift-monitoring/node-exporter-8jhbg" Sep 09 10:30:47.326739 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.326415 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0d829bc-3ccb-4fd7-ab82-bab2b9128c88-metrics-client-ca\") pod \"node-exporter-8jhbg\" (UID: \"e0d829bc-3ccb-4fd7-ab82-bab2b9128c88\") " pod="openshift-monitoring/node-exporter-8jhbg" Sep 09 10:30:47.326739 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.326518 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0d829bc-3ccb-4fd7-ab82-bab2b9128c88-sys\") pod \"node-exporter-8jhbg\" (UID: \"e0d829bc-3ccb-4fd7-ab82-bab2b9128c88\") " pod="openshift-monitoring/node-exporter-8jhbg" Sep 09 10:30:47.326739 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.326635 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e0d829bc-3ccb-4fd7-ab82-bab2b9128c88-node-exporter-wtmp\") pod \"node-exporter-8jhbg\" (UID: \"e0d829bc-3ccb-4fd7-ab82-bab2b9128c88\") " pod="openshift-monitoring/node-exporter-8jhbg" Sep 09 10:30:47.326885 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.326760 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-qnjb6\" (UID: \"3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qnjb6" Sep 09 10:30:47.326885 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.326818 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-qnjb6\" (UID: \"3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qnjb6" Sep 09 10:30:47.328826 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.328755 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e0d829bc-3ccb-4fd7-ab82-bab2b9128c88-node-exporter-tls\") pod \"node-exporter-8jhbg\" (UID: \"e0d829bc-3ccb-4fd7-ab82-bab2b9128c88\") " pod="openshift-monitoring/node-exporter-8jhbg" Sep 09 10:30:47.329083 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.329036 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-qnjb6\" (UID: \"3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qnjb6" Sep 09 10:30:47.329083 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.329050 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0d829bc-3ccb-4fd7-ab82-bab2b9128c88-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8jhbg\" (UID: \"e0d829bc-3ccb-4fd7-ab82-bab2b9128c88\") " pod="openshift-monitoring/node-exporter-8jhbg" Sep 09 10:30:47.329390 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.329372 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-qnjb6\" (UID: \"3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qnjb6" Sep 09 10:30:47.334877 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.334853 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw685\" (UniqueName: \"kubernetes.io/projected/e0d829bc-3ccb-4fd7-ab82-bab2b9128c88-kube-api-access-pw685\") pod \"node-exporter-8jhbg\" (UID: \"e0d829bc-3ccb-4fd7-ab82-bab2b9128c88\") " pod="openshift-monitoring/node-exporter-8jhbg" Sep 09 10:30:47.335071 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.335048 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lj4l\" (UniqueName: \"kubernetes.io/projected/3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3-kube-api-access-2lj4l\") pod \"kube-state-metrics-74cb57df7-qnjb6\" (UID: \"3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qnjb6" Sep 09 10:30:47.418025 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.417987 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8jhbg" Sep 09 10:30:47.425556 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.425526 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-qnjb6" Sep 09 10:30:47.729396 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.729313 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a60cc00-e367-4273-8317-e3e33e871abc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-4p88h\" (UID: \"2a60cc00-e367-4273-8317-e3e33e871abc\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p88h" Sep 09 10:30:47.732071 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.732048 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a60cc00-e367-4273-8317-e3e33e871abc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-4p88h\" (UID: \"2a60cc00-e367-4273-8317-e3e33e871abc\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p88h" Sep 09 10:30:47.988214 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:47.988127 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p88h" Sep 09 10:30:51.525190 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:51.525128 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" podUID="3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 09 10:30:52.338249 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:30:52.338213 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88316610_e58d_4f66_ae6c_eb81674b4f43.slice/crio-70f83a684ae0409dbc4069d0d01b4a54371f47fcff7018582e46d561f4fc8687 WatchSource:0}: Error finding container 70f83a684ae0409dbc4069d0d01b4a54371f47fcff7018582e46d561f4fc8687: Status 404 returned error can't find the container with id 70f83a684ae0409dbc4069d0d01b4a54371f47fcff7018582e46d561f4fc8687 Sep 09 10:30:52.352830 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:30:52.352793 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d829bc_3ccb_4fd7_ab82_bab2b9128c88.slice/crio-714537e91407c4e817ef019888bcb648c980225476b0c9bdae60e142af0c3754 WatchSource:0}: Error finding container 714537e91407c4e817ef019888bcb648c980225476b0c9bdae60e142af0c3754: Status 404 returned error can't find the container with id 714537e91407c4e817ef019888bcb648c980225476b0c9bdae60e142af0c3754 Sep 09 10:30:52.628794 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:52.628597 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-4p88h"] Sep 09 10:30:52.629890 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:52.629858 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-qnjb6"] Sep 09 10:30:52.631921 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:30:52.631886 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a60cc00_e367_4273_8317_e3e33e871abc.slice/crio-a83ed174f324f0f3a39cd16c5af50fc5505d015007c240a25abb49132b811d8c WatchSource:0}: Error finding container a83ed174f324f0f3a39cd16c5af50fc5505d015007c240a25abb49132b811d8c: Status 404 returned error can't find the container with id a83ed174f324f0f3a39cd16c5af50fc5505d015007c240a25abb49132b811d8c Sep 09 10:30:52.632777 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:30:52.632734 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a0e3c69_32c0_4328_8b0e_41c17c0fe6b3.slice/crio-71106851598f1a878f55648677bed405af9da8749701743c0f8886cae409f7e2 WatchSource:0}: Error finding container 71106851598f1a878f55648677bed405af9da8749701743c0f8886cae409f7e2: Status 404 returned error can't find the container with id 71106851598f1a878f55648677bed405af9da8749701743c0f8886cae409f7e2 Sep 09 10:30:53.311198 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:53.311157 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-dhvgm" event={"ID":"b19da99e-eafb-470c-ab9b-fe1e6800a92d","Type":"ContainerStarted","Data":"d7c3059e1be2e991b1cae5bbbdd63436a6fe720541d68d8a287e3c6ea24730b5"} Sep 09 10:30:53.312238 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:53.312215 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-dhvgm" Sep 09 10:30:53.315630 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:53.315594 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-qnjb6" event={"ID":"3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3","Type":"ContainerStarted","Data":"71106851598f1a878f55648677bed405af9da8749701743c0f8886cae409f7e2"} Sep 09 10:30:53.317478 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:53.317258 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8jhbg" event={"ID":"e0d829bc-3ccb-4fd7-ab82-bab2b9128c88","Type":"ContainerStarted","Data":"714537e91407c4e817ef019888bcb648c980225476b0c9bdae60e142af0c3754"} Sep 09 10:30:53.320265 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:53.320200 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jjw92" event={"ID":"88316610-e58d-4f66-ae6c-eb81674b4f43","Type":"ContainerStarted","Data":"70f83a684ae0409dbc4069d0d01b4a54371f47fcff7018582e46d561f4fc8687"} Sep 09 10:30:53.324418 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:53.324298 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p88h" event={"ID":"2a60cc00-e367-4273-8317-e3e33e871abc","Type":"ContainerStarted","Data":"85583177d6a0bafbf8692f67ca519a13a09d335cc14d7666d1f67e81d42a1347"} Sep 09 10:30:53.324418 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:53.324350 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p88h" event={"ID":"2a60cc00-e367-4273-8317-e3e33e871abc","Type":"ContainerStarted","Data":"da8fd5a76dc186fe3a687106782e41382ddcdbca396e8aa196d255093948bcec"} Sep 09 10:30:53.324418 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:53.324380 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p88h" event={"ID":"2a60cc00-e367-4273-8317-e3e33e871abc","Type":"ContainerStarted","Data":"a83ed174f324f0f3a39cd16c5af50fc5505d015007c240a25abb49132b811d8c"} Sep 09 10:30:53.324989 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:53.324938 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-dhvgm" Sep 09 10:30:53.357811 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:53.357424 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-dhvgm" podStartSLOduration=2.06966675 podStartE2EDuration="18.357401193s" totalImagesPullingTime="16.287734443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:53.357072707 +0000 UTC m=+293.531675034" watchObservedRunningTime="2026-09-09 10:30:53.357401193 +0000 UTC m=+293.532003520" Sep 09 10:30:54.331849 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:54.331805 2559 generic.go:356] "Generic (PLEG): container finished" podID="e0d829bc-3ccb-4fd7-ab82-bab2b9128c88" containerID="f5761ef39e58920ce8f7a6b513ba73fb18775db6e1567e12b1f9410d102b6a09" exitCode=0 Sep 09 10:30:54.334777 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:54.334714 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8jhbg" event={"ID":"e0d829bc-3ccb-4fd7-ab82-bab2b9128c88","Type":"ContainerDied","Data":"f5761ef39e58920ce8f7a6b513ba73fb18775db6e1567e12b1f9410d102b6a09"} Sep 09 10:30:56.329547 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:30:56.329472 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-c4s48" podUID="1b8cf871-4dea-4ac5-868a-35b02f455e65" Sep 09 10:30:56.342726 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:56.342686 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-qnjb6" event={"ID":"3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3","Type":"ContainerStarted","Data":"25943b2656881fbff632062202a1571fd7571d4b70938ed95ebd813824966c60"} Sep 09 10:30:56.342895 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:56.342757 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-qnjb6" event={"ID":"3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3","Type":"ContainerStarted","Data":"bbb359be4b3c4aa4b662d0d1d40077e8637b05c785012d8faf7abda47d130e19"} Sep 09 10:30:56.342895 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:56.342790 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-qnjb6" event={"ID":"3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3","Type":"ContainerStarted","Data":"8bbfef4e9c10e200a3fd58c2dbe0ed0f4107f8a1bd3550204113587c4fb14129"} Sep 09 10:30:56.345562 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:56.345530 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8jhbg" event={"ID":"e0d829bc-3ccb-4fd7-ab82-bab2b9128c88","Type":"ContainerStarted","Data":"a7affbb7235e1d748f8bb8d5d35f20f119718e9f13410fc1f99a5e32852793a2"} Sep 09 10:30:56.345696 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:56.345588 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8jhbg" event={"ID":"e0d829bc-3ccb-4fd7-ab82-bab2b9128c88","Type":"ContainerStarted","Data":"33b4804fbbd6cdd21833acc03159ae73efca6a59a64edf98abceb80fa167b92c"} Sep 09 10:30:56.347795 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:56.347764 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jjw92" event={"ID":"88316610-e58d-4f66-ae6c-eb81674b4f43","Type":"ContainerStarted","Data":"62bd6e083ac8dd04c0979b9493a20b32fbfa331c67599790c176bfffca5dd4f4"} Sep 09 10:30:56.347911 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:56.347818 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jjw92" event={"ID":"88316610-e58d-4f66-ae6c-eb81674b4f43","Type":"ContainerStarted","Data":"fd64d062594dcb59b3ac2eb2d9aca89c1d5e19a2c455806a70e71084a00a0286"} Sep 09 10:30:56.348319 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:56.348297 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jjw92" Sep 09 10:30:56.350490 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:56.350412 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p88h" event={"ID":"2a60cc00-e367-4273-8317-e3e33e871abc","Type":"ContainerStarted","Data":"3ce84c3e6eb036cbb914a605a79ff5234a1368b8c36e32549a55453a47256655"} Sep 09 10:30:56.365535 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:56.365462 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-qnjb6" podStartSLOduration=6.169534191 podStartE2EDuration="9.365444726s" totalImagesPullingTime="3.195910535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:30:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:56.364255056 +0000 UTC m=+296.538857382" watchObservedRunningTime="2026-09-09 10:30:56.365444726 +0000 UTC m=+296.540047055" Sep 09 10:30:56.382254 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:56.382146 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p88h" podStartSLOduration=6.310860933 podStartE2EDuration="9.382129427s" totalImagesPullingTime="3.071268494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:30:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:56.381839293 +0000 UTC m=+296.556441619" watchObservedRunningTime="2026-09-09 10:30:56.382129427 +0000 UTC m=+296.556731753" Sep 09 10:30:56.399244 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:56.399177 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jjw92" podStartSLOduration=257.905737891 podStartE2EDuration="4m21.399160305s" totalImagesPullingTime="3.493422414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:26:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:56.39898741 +0000 UTC m=+296.573589736" watchObservedRunningTime="2026-09-09 10:30:56.399160305 +0000 UTC m=+296.573762630" Sep 09 10:30:56.417293 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:30:56.417225 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8jhbg" podStartSLOduration=7.984236532 podStartE2EDuration="9.417202024s" totalImagesPullingTime="1.3457083s" totalInitContainerRuntime="87.257192ms" isStatefulPod=true podCreationTimestamp="2026-09-09 10:30:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:30:56.416663705 +0000 UTC m=+296.591266031" watchObservedRunningTime="2026-09-09 10:30:56.417202024 +0000 UTC m=+296.591804352" Sep 09 10:31:00.221833 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:31:00.221794 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d22ff_0a56b285-3f3c-4f6d-a939-8a95ae00e8da/ovn-acl-logging/0.log" Sep 09 10:31:00.222654 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:31:00.222630 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d22ff_0a56b285-3f3c-4f6d-a939-8a95ae00e8da/ovn-acl-logging/0.log" Sep 09 10:31:00.232030 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:31:00.232010 2559 kubelet.go:1648] "Image garbage collection succeeded" Sep 09 10:31:01.525190 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:31:01.525147 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" podUID="3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 09 10:31:06.357603 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:31:06.357565 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jjw92" Sep 09 10:31:07.328853 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:31:07.328808 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:31:11.524663 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:31:11.524621 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" podUID="3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 09 10:31:11.525121 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:31:11.524717 2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" Sep 09 10:31:11.525222 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:31:11.525191 2559 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"ebfe7883c08a5c081d125ad8cc10eba419db4768e01028a964035b21d86efa7f"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" containerMessage="Container service-proxy failed liveness probe, will be restarted" Sep 09 10:31:11.525267 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:31:11.525242 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" podUID="3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a" containerName="service-proxy" containerID="cri-o://ebfe7883c08a5c081d125ad8cc10eba419db4768e01028a964035b21d86efa7f" gracePeriod=30 Sep 09 10:31:12.417698 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:31:12.417663 2559 generic.go:356] "Generic (PLEG): container finished" podID="3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a" containerID="ebfe7883c08a5c081d125ad8cc10eba419db4768e01028a964035b21d86efa7f" exitCode=2 Sep 09 10:31:12.417885 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:31:12.417734 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" event={"ID":"3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a","Type":"ContainerDied","Data":"ebfe7883c08a5c081d125ad8cc10eba419db4768e01028a964035b21d86efa7f"} Sep 09 10:31:12.417885 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:31:12.417778 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f9d66b65f-xn58h" event={"ID":"3384b0aa-c7d3-4e1c-b73f-6b96a924ed7a","Type":"ContainerStarted","Data":"cdc25d57f4ba947ce2a882320e200fae30bdf040155419f9bfaaa879952ed0c1"} Sep 09 10:32:13.216441 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:32:13.216379 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs\") pod \"network-metrics-daemon-c4s48\" (UID: \"1b8cf871-4dea-4ac5-868a-35b02f455e65\") " pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:32:13.218756 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:32:13.218732 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b8cf871-4dea-4ac5-868a-35b02f455e65-metrics-certs\") pod \"network-metrics-daemon-c4s48\" (UID: \"1b8cf871-4dea-4ac5-868a-35b02f455e65\") " pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:32:13.332598 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:32:13.332565 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hq5pc\"" Sep 09 10:32:13.339979 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:32:13.339957 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4s48" Sep 09 10:32:13.607837 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:32:13.607806 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c4s48"] Sep 09 10:32:13.610962 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:32:13.610931 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b8cf871_4dea_4ac5_868a_35b02f455e65.slice/crio-5ff98c63156c5b3a12e37ec2490a22aeffc01ab6b8ca76a3a8103071e9634d54 WatchSource:0}: Error finding container 5ff98c63156c5b3a12e37ec2490a22aeffc01ab6b8ca76a3a8103071e9634d54: Status 404 returned error can't find the container with id 5ff98c63156c5b3a12e37ec2490a22aeffc01ab6b8ca76a3a8103071e9634d54 Sep 09 10:32:13.612989 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:32:13.612974 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 10:32:13.637395 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:32:13.637363 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c4s48" event={"ID":"1b8cf871-4dea-4ac5-868a-35b02f455e65","Type":"ContainerStarted","Data":"5ff98c63156c5b3a12e37ec2490a22aeffc01ab6b8ca76a3a8103071e9634d54"} Sep 09 10:32:15.646476 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:32:15.646429 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c4s48" event={"ID":"1b8cf871-4dea-4ac5-868a-35b02f455e65","Type":"ContainerStarted","Data":"024a602b5c416d032723ff6457a94214546b6a9d82e4a003ac923fdd270d7413"} Sep 09 10:32:15.646872 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:32:15.646496 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c4s48" event={"ID":"1b8cf871-4dea-4ac5-868a-35b02f455e65","Type":"ContainerStarted","Data":"5e7bcaee7e1f8429e3c2d2779d0ad78d34290ae774a76a9f5831ac96805a3242"} Sep 09 10:32:15.669238 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:32:15.669183 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-c4s48" podStartSLOduration=374.361083444 podStartE2EDuration="6m15.669165814s" totalImagesPullingTime="1.30808237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:32:15.667869851 +0000 UTC m=+375.842472177" watchObservedRunningTime="2026-09-09 10:32:15.669165814 +0000 UTC m=+375.843768139" Sep 09 10:36:00.245423 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:00.245392 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d22ff_0a56b285-3f3c-4f6d-a939-8a95ae00e8da/ovn-acl-logging/0.log" Sep 09 10:36:00.247464 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:00.247443 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d22ff_0a56b285-3f3c-4f6d-a939-8a95ae00e8da/ovn-acl-logging/0.log" Sep 09 10:36:20.488958 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:20.488924 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-bgrxh"] Sep 09 10:36:20.490843 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:20.490822 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-bgrxh" Sep 09 10:36:20.493419 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:20.493394 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 09 10:36:20.493528 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:20.493420 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-vdv9z\"" Sep 09 10:36:20.494269 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:20.494254 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 09 10:36:20.502054 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:20.502035 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-bgrxh"] Sep 09 10:36:20.650801 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:20.650767 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4189bcda-a610-448e-80da-06abca93520f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-bgrxh\" (UID: \"4189bcda-a610-448e-80da-06abca93520f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bgrxh" Sep 09 10:36:20.650965 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:20.650808 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh5l5\" (UniqueName: \"kubernetes.io/projected/4189bcda-a610-448e-80da-06abca93520f-kube-api-access-fh5l5\") pod \"cert-manager-webhook-6486ffd997-bgrxh\" (UID: \"4189bcda-a610-448e-80da-06abca93520f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bgrxh" Sep 09 10:36:20.751324 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:20.751237 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4189bcda-a610-448e-80da-06abca93520f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-bgrxh\" (UID: \"4189bcda-a610-448e-80da-06abca93520f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bgrxh" Sep 09 10:36:20.751324 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:20.751283 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh5l5\" (UniqueName: \"kubernetes.io/projected/4189bcda-a610-448e-80da-06abca93520f-kube-api-access-fh5l5\") pod \"cert-manager-webhook-6486ffd997-bgrxh\" (UID: \"4189bcda-a610-448e-80da-06abca93520f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bgrxh" Sep 09 10:36:20.759922 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:20.759888 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh5l5\" (UniqueName: \"kubernetes.io/projected/4189bcda-a610-448e-80da-06abca93520f-kube-api-access-fh5l5\") pod \"cert-manager-webhook-6486ffd997-bgrxh\" (UID: \"4189bcda-a610-448e-80da-06abca93520f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bgrxh" Sep 09 10:36:20.760162 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:20.760142 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4189bcda-a610-448e-80da-06abca93520f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-bgrxh\" (UID: \"4189bcda-a610-448e-80da-06abca93520f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bgrxh" Sep 09 10:36:20.799937 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:20.799899 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-bgrxh" Sep 09 10:36:21.070513 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:21.068762 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-bgrxh"] Sep 09 10:36:21.074598 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:36:21.074560 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4189bcda_a610_448e_80da_06abca93520f.slice/crio-89c3d9a93debcc0a87c25dafae673ebe6bb1952bbd3716cfba3335ebf017bb0d WatchSource:0}: Error finding container 89c3d9a93debcc0a87c25dafae673ebe6bb1952bbd3716cfba3335ebf017bb0d: Status 404 returned error can't find the container with id 89c3d9a93debcc0a87c25dafae673ebe6bb1952bbd3716cfba3335ebf017bb0d Sep 09 10:36:21.498628 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:21.498581 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-bgrxh" event={"ID":"4189bcda-a610-448e-80da-06abca93520f","Type":"ContainerStarted","Data":"89c3d9a93debcc0a87c25dafae673ebe6bb1952bbd3716cfba3335ebf017bb0d"} Sep 09 10:36:24.515016 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:24.514985 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-bgrxh" event={"ID":"4189bcda-a610-448e-80da-06abca93520f","Type":"ContainerStarted","Data":"a088b276a882fc8700095a760376fc991075f7155875bc9a172d06c7d2649e05"} Sep 09 10:36:24.515447 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:24.515205 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-bgrxh" Sep 09 10:36:24.534314 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:24.534258 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-bgrxh" podStartSLOduration=1.8913067890000002 podStartE2EDuration="4.534238789s" totalImagesPullingTime="2.642932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:36:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:36:24.533445041 +0000 UTC m=+624.708047366" watchObservedRunningTime="2026-09-09 10:36:24.534238789 +0000 UTC m=+624.708841115" Sep 09 10:36:30.521612 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:30.521579 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-bgrxh" Sep 09 10:36:34.624697 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:34.624660 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7ln7j"] Sep 09 10:36:34.626862 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:34.626845 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-7ln7j" Sep 09 10:36:34.629581 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:34.629545 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-t2bg9\"" Sep 09 10:36:34.638087 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:34.638062 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7ln7j"] Sep 09 10:36:34.654152 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:34.654123 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm46x\" (UniqueName: \"kubernetes.io/projected/0d664150-aa03-4d4b-8ef6-ea79d83ac66c-kube-api-access-wm46x\") pod \"cert-manager-55b4d954d5-7ln7j\" (UID: \"0d664150-aa03-4d4b-8ef6-ea79d83ac66c\") " pod="cert-manager/cert-manager-55b4d954d5-7ln7j" Sep 09 10:36:34.654291 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:34.654158 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d664150-aa03-4d4b-8ef6-ea79d83ac66c-bound-sa-token\") pod \"cert-manager-55b4d954d5-7ln7j\" (UID: \"0d664150-aa03-4d4b-8ef6-ea79d83ac66c\") " pod="cert-manager/cert-manager-55b4d954d5-7ln7j" Sep 09 10:36:34.755288 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:34.755246 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm46x\" (UniqueName: \"kubernetes.io/projected/0d664150-aa03-4d4b-8ef6-ea79d83ac66c-kube-api-access-wm46x\") pod \"cert-manager-55b4d954d5-7ln7j\" (UID: \"0d664150-aa03-4d4b-8ef6-ea79d83ac66c\") " pod="cert-manager/cert-manager-55b4d954d5-7ln7j" Sep 09 10:36:34.755288 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:34.755293 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d664150-aa03-4d4b-8ef6-ea79d83ac66c-bound-sa-token\") pod \"cert-manager-55b4d954d5-7ln7j\" (UID: \"0d664150-aa03-4d4b-8ef6-ea79d83ac66c\") " pod="cert-manager/cert-manager-55b4d954d5-7ln7j" Sep 09 10:36:34.764722 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:34.764692 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d664150-aa03-4d4b-8ef6-ea79d83ac66c-bound-sa-token\") pod \"cert-manager-55b4d954d5-7ln7j\" (UID: \"0d664150-aa03-4d4b-8ef6-ea79d83ac66c\") " pod="cert-manager/cert-manager-55b4d954d5-7ln7j" Sep 09 10:36:34.764861 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:34.764842 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm46x\" (UniqueName: \"kubernetes.io/projected/0d664150-aa03-4d4b-8ef6-ea79d83ac66c-kube-api-access-wm46x\") pod \"cert-manager-55b4d954d5-7ln7j\" (UID: \"0d664150-aa03-4d4b-8ef6-ea79d83ac66c\") " pod="cert-manager/cert-manager-55b4d954d5-7ln7j" Sep 09 10:36:34.935415 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:34.935324 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-7ln7j" Sep 09 10:36:35.205358 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:35.205283 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7ln7j"] Sep 09 10:36:35.208289 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:36:35.208261 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d664150_aa03_4d4b_8ef6_ea79d83ac66c.slice/crio-be753c0b832c73357c7f1c103bf833b85d546b506d507598be8d3d6d5bdfeb7b WatchSource:0}: Error finding container be753c0b832c73357c7f1c103bf833b85d546b506d507598be8d3d6d5bdfeb7b: Status 404 returned error can't find the container with id be753c0b832c73357c7f1c103bf833b85d546b506d507598be8d3d6d5bdfeb7b Sep 09 10:36:35.554799 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:35.554706 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-7ln7j" event={"ID":"0d664150-aa03-4d4b-8ef6-ea79d83ac66c","Type":"ContainerStarted","Data":"a44b02a4a29995e790d8eb60c30e253f6d8c611a1d5d2ca950b05c071e082d75"} Sep 09 10:36:35.554799 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:35.554752 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-7ln7j" event={"ID":"0d664150-aa03-4d4b-8ef6-ea79d83ac66c","Type":"ContainerStarted","Data":"be753c0b832c73357c7f1c103bf833b85d546b506d507598be8d3d6d5bdfeb7b"} Sep 09 10:36:35.571513 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:35.571447 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-7ln7j" podStartSLOduration=1.5714313450000001 podStartE2EDuration="1.571431345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:36:35.570776497 +0000 UTC m=+635.745378822" watchObservedRunningTime="2026-09-09 10:36:35.571431345 +0000 UTC m=+635.746033669" Sep 09 10:36:51.597392 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:51.597349 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2nr65"] Sep 09 10:36:51.599876 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:51.599854 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2nr65" Sep 09 10:36:51.605193 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:51.605165 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 09 10:36:51.605571 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:51.605550 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 09 10:36:51.606616 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:51.606595 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 09 10:36:51.606682 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:51.606596 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 09 10:36:51.607241 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:51.607223 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-llqpt\"" Sep 09 10:36:51.619337 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:51.619312 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2nr65"] Sep 09 10:36:51.679940 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:51.679905 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2658a6e-38ca-405b-bb5c-5767972f0fc9-tmp\") pod \"opendatahub-operator-controller-manager-544cbf49fb-2nr65\" (UID: \"a2658a6e-38ca-405b-bb5c-5767972f0fc9\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2nr65" Sep 09 10:36:51.679940 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:51.679940 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a2658a6e-38ca-405b-bb5c-5767972f0fc9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-544cbf49fb-2nr65\" (UID: \"a2658a6e-38ca-405b-bb5c-5767972f0fc9\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2nr65" Sep 09 10:36:51.680152 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:51.679972 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2658a6e-38ca-405b-bb5c-5767972f0fc9-webhook-cert\") pod \"opendatahub-operator-controller-manager-544cbf49fb-2nr65\" (UID: \"a2658a6e-38ca-405b-bb5c-5767972f0fc9\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2nr65" Sep 09 10:36:51.680152 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:51.680048 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a2658a6e-38ca-405b-bb5c-5767972f0fc9-manifests\") pod \"opendatahub-operator-controller-manager-544cbf49fb-2nr65\" (UID: \"a2658a6e-38ca-405b-bb5c-5767972f0fc9\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2nr65" Sep 09 10:36:51.680152 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:51.680096 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkn6s\" (UniqueName: \"kubernetes.io/projected/a2658a6e-38ca-405b-bb5c-5767972f0fc9-kube-api-access-gkn6s\") pod \"opendatahub-operator-controller-manager-544cbf49fb-2nr65\" (UID: \"a2658a6e-38ca-405b-bb5c-5767972f0fc9\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2nr65" Sep 09 10:36:51.780590 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:51.780548 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2658a6e-38ca-405b-bb5c-5767972f0fc9-tmp\") pod \"opendatahub-operator-controller-manager-544cbf49fb-2nr65\" (UID: \"a2658a6e-38ca-405b-bb5c-5767972f0fc9\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2nr65" Sep 09 10:36:51.780590 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:51.780593 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a2658a6e-38ca-405b-bb5c-5767972f0fc9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-544cbf49fb-2nr65\" (UID: \"a2658a6e-38ca-405b-bb5c-5767972f0fc9\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2nr65" Sep 09 10:36:51.780819 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:51.780626 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2658a6e-38ca-405b-bb5c-5767972f0fc9-webhook-cert\") pod \"opendatahub-operator-controller-manager-544cbf49fb-2nr65\" (UID: \"a2658a6e-38ca-405b-bb5c-5767972f0fc9\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2nr65" Sep 09 10:36:51.780819 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:51.780646 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a2658a6e-38ca-405b-bb5c-5767972f0fc9-manifests\") pod \"opendatahub-operator-controller-manager-544cbf49fb-2nr65\" (UID: \"a2658a6e-38ca-405b-bb5c-5767972f0fc9\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2nr65" Sep 09 10:36:51.780819 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:51.780663 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkn6s\" (UniqueName: \"kubernetes.io/projected/a2658a6e-38ca-405b-bb5c-5767972f0fc9-kube-api-access-gkn6s\") pod \"opendatahub-operator-controller-manager-544cbf49fb-2nr65\" (UID: \"a2658a6e-38ca-405b-bb5c-5767972f0fc9\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2nr65" Sep 09 10:36:51.781092 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:51.781071 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a2658a6e-38ca-405b-bb5c-5767972f0fc9-manifests\") pod \"opendatahub-operator-controller-manager-544cbf49fb-2nr65\" (UID: \"a2658a6e-38ca-405b-bb5c-5767972f0fc9\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2nr65" Sep 09 10:36:51.783059 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:51.783027 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2658a6e-38ca-405b-bb5c-5767972f0fc9-tmp\") pod \"opendatahub-operator-controller-manager-544cbf49fb-2nr65\" (UID: \"a2658a6e-38ca-405b-bb5c-5767972f0fc9\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2nr65" Sep 09 10:36:51.783183 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:51.783154 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a2658a6e-38ca-405b-bb5c-5767972f0fc9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-544cbf49fb-2nr65\" (UID: \"a2658a6e-38ca-405b-bb5c-5767972f0fc9\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2nr65" Sep 09 10:36:51.783275 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:51.783256 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2658a6e-38ca-405b-bb5c-5767972f0fc9-webhook-cert\") pod \"opendatahub-operator-controller-manager-544cbf49fb-2nr65\" (UID: \"a2658a6e-38ca-405b-bb5c-5767972f0fc9\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2nr65" Sep 09 10:36:51.790441 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:51.790398 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkn6s\" (UniqueName: \"kubernetes.io/projected/a2658a6e-38ca-405b-bb5c-5767972f0fc9-kube-api-access-gkn6s\") pod \"opendatahub-operator-controller-manager-544cbf49fb-2nr65\" (UID: \"a2658a6e-38ca-405b-bb5c-5767972f0fc9\") " pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2nr65" Sep 09 10:36:51.911150 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:51.911056 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2nr65" Sep 09 10:36:52.192165 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:52.192086 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2nr65"] Sep 09 10:36:52.196718 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:36:52.196686 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2658a6e_38ca_405b_bb5c_5767972f0fc9.slice/crio-92bcb52a147fc1a9cd0357f63df31f495095a7a9ecd1319f27b611c00072e740 WatchSource:0}: Error finding container 92bcb52a147fc1a9cd0357f63df31f495095a7a9ecd1319f27b611c00072e740: Status 404 returned error can't find the container with id 92bcb52a147fc1a9cd0357f63df31f495095a7a9ecd1319f27b611c00072e740 Sep 09 10:36:52.617999 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:52.617963 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2nr65" event={"ID":"a2658a6e-38ca-405b-bb5c-5767972f0fc9","Type":"ContainerStarted","Data":"92bcb52a147fc1a9cd0357f63df31f495095a7a9ecd1319f27b611c00072e740"} Sep 09 10:36:55.632683 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:55.632647 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2nr65" event={"ID":"a2658a6e-38ca-405b-bb5c-5767972f0fc9","Type":"ContainerStarted","Data":"09481678d317ebbf4665315568d9484a066760717a12d7d6e378a1bff6ae45f6"} Sep 09 10:36:56.638355 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:56.638322 2559 generic.go:356] "Generic (PLEG): container finished" podID="a2658a6e-38ca-405b-bb5c-5767972f0fc9" containerID="09481678d317ebbf4665315568d9484a066760717a12d7d6e378a1bff6ae45f6" exitCode=0 Sep 09 10:36:56.638772 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:56.638417 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2nr65" event={"ID":"a2658a6e-38ca-405b-bb5c-5767972f0fc9","Type":"ContainerDied","Data":"09481678d317ebbf4665315568d9484a066760717a12d7d6e378a1bff6ae45f6"} Sep 09 10:36:57.645469 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:57.645433 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2nr65" event={"ID":"a2658a6e-38ca-405b-bb5c-5767972f0fc9","Type":"ContainerStarted","Data":"67d7211a97342cbc7c779c754927c8300e4383b419875b9c9c6e3865dadca316"} Sep 09 10:36:57.645878 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:57.645538 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2nr65" Sep 09 10:36:57.670127 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:36:57.670057 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2nr65" podStartSLOduration=2.24197203 podStartE2EDuration="6.67003095s" totalImagesPullingTime="3.370553339s" totalInitContainerRuntime="1.057505581s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:36:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:36:57.667905808 +0000 UTC m=+657.842508147" watchObservedRunningTime="2026-09-09 10:36:57.67003095 +0000 UTC m=+657.844633276" Sep 09 10:37:08.651261 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:08.651227 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-544cbf49fb-2nr65" Sep 09 10:37:12.257978 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:12.257939 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-75cd99c5fc-fqj8r"] Sep 09 10:37:12.264748 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:12.264728 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-75cd99c5fc-fqj8r" Sep 09 10:37:12.284074 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:12.284040 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 09 10:37:12.293868 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:12.293839 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-75cd99c5fc-fqj8r"] Sep 09 10:37:12.302969 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:12.302933 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-489j2\"" Sep 09 10:37:12.303146 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:12.303120 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 09 10:37:12.303146 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:12.303120 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 09 10:37:12.303254 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:12.303218 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 09 10:37:12.345802 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:12.345764 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6wft\" (UniqueName: \"kubernetes.io/projected/c719a5cf-09c3-4950-916a-33922d99dad4-kube-api-access-h6wft\") pod \"kube-auth-proxy-75cd99c5fc-fqj8r\" (UID: \"c719a5cf-09c3-4950-916a-33922d99dad4\") " pod="openshift-ingress/kube-auth-proxy-75cd99c5fc-fqj8r" Sep 09 10:37:12.345992 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:12.345814 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c719a5cf-09c3-4950-916a-33922d99dad4-tls-certs\") pod \"kube-auth-proxy-75cd99c5fc-fqj8r\" (UID: \"c719a5cf-09c3-4950-916a-33922d99dad4\") " pod="openshift-ingress/kube-auth-proxy-75cd99c5fc-fqj8r" Sep 09 10:37:12.345992 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:12.345836 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c719a5cf-09c3-4950-916a-33922d99dad4-tmp\") pod \"kube-auth-proxy-75cd99c5fc-fqj8r\" (UID: \"c719a5cf-09c3-4950-916a-33922d99dad4\") " pod="openshift-ingress/kube-auth-proxy-75cd99c5fc-fqj8r" Sep 09 10:37:12.447065 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:12.447030 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6wft\" (UniqueName: \"kubernetes.io/projected/c719a5cf-09c3-4950-916a-33922d99dad4-kube-api-access-h6wft\") pod \"kube-auth-proxy-75cd99c5fc-fqj8r\" (UID: \"c719a5cf-09c3-4950-916a-33922d99dad4\") " pod="openshift-ingress/kube-auth-proxy-75cd99c5fc-fqj8r" Sep 09 10:37:12.447235 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:12.447091 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c719a5cf-09c3-4950-916a-33922d99dad4-tls-certs\") pod \"kube-auth-proxy-75cd99c5fc-fqj8r\" (UID: \"c719a5cf-09c3-4950-916a-33922d99dad4\") " pod="openshift-ingress/kube-auth-proxy-75cd99c5fc-fqj8r" Sep 09 10:37:12.447235 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:12.447115 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c719a5cf-09c3-4950-916a-33922d99dad4-tmp\") pod \"kube-auth-proxy-75cd99c5fc-fqj8r\" (UID: \"c719a5cf-09c3-4950-916a-33922d99dad4\") " pod="openshift-ingress/kube-auth-proxy-75cd99c5fc-fqj8r" Sep 09 10:37:12.449492 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:12.449463 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c719a5cf-09c3-4950-916a-33922d99dad4-tmp\") pod \"kube-auth-proxy-75cd99c5fc-fqj8r\" (UID: \"c719a5cf-09c3-4950-916a-33922d99dad4\") " pod="openshift-ingress/kube-auth-proxy-75cd99c5fc-fqj8r" Sep 09 10:37:12.449721 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:12.449703 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c719a5cf-09c3-4950-916a-33922d99dad4-tls-certs\") pod \"kube-auth-proxy-75cd99c5fc-fqj8r\" (UID: \"c719a5cf-09c3-4950-916a-33922d99dad4\") " pod="openshift-ingress/kube-auth-proxy-75cd99c5fc-fqj8r" Sep 09 10:37:12.464070 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:12.464038 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6wft\" (UniqueName: \"kubernetes.io/projected/c719a5cf-09c3-4950-916a-33922d99dad4-kube-api-access-h6wft\") pod \"kube-auth-proxy-75cd99c5fc-fqj8r\" (UID: \"c719a5cf-09c3-4950-916a-33922d99dad4\") " pod="openshift-ingress/kube-auth-proxy-75cd99c5fc-fqj8r" Sep 09 10:37:12.573648 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:12.573555 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-75cd99c5fc-fqj8r" Sep 09 10:37:12.848615 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:12.848584 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-75cd99c5fc-fqj8r"] Sep 09 10:37:12.852144 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:37:12.852119 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc719a5cf_09c3_4950_916a_33922d99dad4.slice/crio-6400814f33cb4d43a18bd279d206b9144b1f46003248e5646d76279c7a8edf5f WatchSource:0}: Error finding container 6400814f33cb4d43a18bd279d206b9144b1f46003248e5646d76279c7a8edf5f: Status 404 returned error can't find the container with id 6400814f33cb4d43a18bd279d206b9144b1f46003248e5646d76279c7a8edf5f Sep 09 10:37:13.706811 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:13.706767 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-75cd99c5fc-fqj8r" event={"ID":"c719a5cf-09c3-4950-916a-33922d99dad4","Type":"ContainerStarted","Data":"6400814f33cb4d43a18bd279d206b9144b1f46003248e5646d76279c7a8edf5f"} Sep 09 10:37:15.717449 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:15.717415 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-75cd99c5fc-fqj8r" event={"ID":"c719a5cf-09c3-4950-916a-33922d99dad4","Type":"ContainerStarted","Data":"32fa174974da7fa1732806eceb1d62386360e3f0e28b388370005ae58cf42407"} Sep 09 10:37:15.737454 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:15.737398 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-75cd99c5fc-fqj8r" podStartSLOduration=0.960299899 podStartE2EDuration="3.737381592s" totalImagesPullingTime="2.777081693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:37:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:37:15.736693883 +0000 UTC m=+675.911296208" watchObservedRunningTime="2026-09-09 10:37:15.737381592 +0000 UTC m=+675.911983916" Sep 09 10:37:29.757649 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:29.757614 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-v2t7d"] Sep 09 10:37:29.762843 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:29.762822 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v2t7d" Sep 09 10:37:29.766715 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:29.766697 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 09 10:37:29.766818 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:29.766696 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 09 10:37:29.767225 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:29.767207 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 09 10:37:29.767288 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:29.767252 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-r4662\"" Sep 09 10:37:29.767288 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:29.767278 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 09 10:37:29.778881 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:29.778855 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-v2t7d"] Sep 09 10:37:29.885816 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:29.885785 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af4f8698-6435-4c45-8a43-0d2eba16ce43-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-v2t7d\" (UID: \"af4f8698-6435-4c45-8a43-0d2eba16ce43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v2t7d" Sep 09 10:37:29.885961 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:29.885867 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/af4f8698-6435-4c45-8a43-0d2eba16ce43-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-v2t7d\" (UID: \"af4f8698-6435-4c45-8a43-0d2eba16ce43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v2t7d" Sep 09 10:37:29.885961 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:29.885892 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af4f8698-6435-4c45-8a43-0d2eba16ce43-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-v2t7d\" (UID: \"af4f8698-6435-4c45-8a43-0d2eba16ce43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v2t7d" Sep 09 10:37:29.885961 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:29.885912 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/af4f8698-6435-4c45-8a43-0d2eba16ce43-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-v2t7d\" (UID: \"af4f8698-6435-4c45-8a43-0d2eba16ce43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v2t7d" Sep 09 10:37:29.885961 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:29.885931 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/af4f8698-6435-4c45-8a43-0d2eba16ce43-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-v2t7d\" (UID: \"af4f8698-6435-4c45-8a43-0d2eba16ce43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v2t7d" Sep 09 10:37:29.886104 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:29.886004 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l88jt\" (UniqueName: \"kubernetes.io/projected/af4f8698-6435-4c45-8a43-0d2eba16ce43-kube-api-access-l88jt\") pod \"istiod-openshift-gateway-5677ccb49b-v2t7d\" (UID: \"af4f8698-6435-4c45-8a43-0d2eba16ce43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v2t7d" Sep 09 10:37:29.886104 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:29.886049 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/af4f8698-6435-4c45-8a43-0d2eba16ce43-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-v2t7d\" (UID: \"af4f8698-6435-4c45-8a43-0d2eba16ce43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v2t7d" Sep 09 10:37:29.987218 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:29.987187 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af4f8698-6435-4c45-8a43-0d2eba16ce43-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-v2t7d\" (UID: \"af4f8698-6435-4c45-8a43-0d2eba16ce43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v2t7d" Sep 09 10:37:29.987218 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:29.987222 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/af4f8698-6435-4c45-8a43-0d2eba16ce43-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-v2t7d\" (UID: \"af4f8698-6435-4c45-8a43-0d2eba16ce43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v2t7d" Sep 09 10:37:29.987391 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:29.987241 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/af4f8698-6435-4c45-8a43-0d2eba16ce43-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-v2t7d\" (UID: \"af4f8698-6435-4c45-8a43-0d2eba16ce43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v2t7d" Sep 09 10:37:29.987391 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:29.987350 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l88jt\" (UniqueName: \"kubernetes.io/projected/af4f8698-6435-4c45-8a43-0d2eba16ce43-kube-api-access-l88jt\") pod \"istiod-openshift-gateway-5677ccb49b-v2t7d\" (UID: \"af4f8698-6435-4c45-8a43-0d2eba16ce43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v2t7d" Sep 09 10:37:29.987495 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:29.987395 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/af4f8698-6435-4c45-8a43-0d2eba16ce43-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-v2t7d\" (UID: \"af4f8698-6435-4c45-8a43-0d2eba16ce43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v2t7d" Sep 09 10:37:29.987495 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:29.987432 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af4f8698-6435-4c45-8a43-0d2eba16ce43-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-v2t7d\" (UID: \"af4f8698-6435-4c45-8a43-0d2eba16ce43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v2t7d" Sep 09 10:37:29.987495 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:29.987479 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/af4f8698-6435-4c45-8a43-0d2eba16ce43-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-v2t7d\" (UID: \"af4f8698-6435-4c45-8a43-0d2eba16ce43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v2t7d" Sep 09 10:37:29.988239 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:29.988215 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/af4f8698-6435-4c45-8a43-0d2eba16ce43-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-v2t7d\" (UID: \"af4f8698-6435-4c45-8a43-0d2eba16ce43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v2t7d" Sep 09 10:37:29.989652 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:29.989625 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af4f8698-6435-4c45-8a43-0d2eba16ce43-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-v2t7d\" (UID: \"af4f8698-6435-4c45-8a43-0d2eba16ce43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v2t7d" Sep 09 10:37:29.989776 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:29.989750 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/af4f8698-6435-4c45-8a43-0d2eba16ce43-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-v2t7d\" (UID: \"af4f8698-6435-4c45-8a43-0d2eba16ce43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v2t7d" Sep 09 10:37:29.989907 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:29.989888 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/af4f8698-6435-4c45-8a43-0d2eba16ce43-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-v2t7d\" (UID: \"af4f8698-6435-4c45-8a43-0d2eba16ce43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v2t7d" Sep 09 10:37:29.989944 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:29.989928 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/af4f8698-6435-4c45-8a43-0d2eba16ce43-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-v2t7d\" (UID: \"af4f8698-6435-4c45-8a43-0d2eba16ce43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v2t7d" Sep 09 10:37:29.995966 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:29.995944 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af4f8698-6435-4c45-8a43-0d2eba16ce43-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-v2t7d\" (UID: \"af4f8698-6435-4c45-8a43-0d2eba16ce43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v2t7d" Sep 09 10:37:29.996141 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:29.996124 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l88jt\" (UniqueName: \"kubernetes.io/projected/af4f8698-6435-4c45-8a43-0d2eba16ce43-kube-api-access-l88jt\") pod \"istiod-openshift-gateway-5677ccb49b-v2t7d\" (UID: \"af4f8698-6435-4c45-8a43-0d2eba16ce43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v2t7d" Sep 09 10:37:30.072164 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:30.072093 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v2t7d" Sep 09 10:37:30.400735 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:30.400674 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-v2t7d"] Sep 09 10:37:30.405549 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:37:30.405515 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf4f8698_6435_4c45_8a43_0d2eba16ce43.slice/crio-c8b3acd4517b3f25e2feb64f3cd9ecf7ff6d5b571440ac299e63de7668f4d50d WatchSource:0}: Error finding container c8b3acd4517b3f25e2feb64f3cd9ecf7ff6d5b571440ac299e63de7668f4d50d: Status 404 returned error can't find the container with id c8b3acd4517b3f25e2feb64f3cd9ecf7ff6d5b571440ac299e63de7668f4d50d Sep 09 10:37:30.407898 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:30.407879 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 10:37:30.783442 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:30.783358 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v2t7d" event={"ID":"af4f8698-6435-4c45-8a43-0d2eba16ce43","Type":"ContainerStarted","Data":"c8b3acd4517b3f25e2feb64f3cd9ecf7ff6d5b571440ac299e63de7668f4d50d"} Sep 09 10:37:32.857865 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:32.857822 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-k45xt"] Sep 09 10:37:32.860914 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:32.860897 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-k45xt" Sep 09 10:37:32.864556 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:32.864530 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 09 10:37:32.864698 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:32.864568 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-22gnj\"" Sep 09 10:37:32.865051 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:32.865030 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-xd7gg"] Sep 09 10:37:32.867251 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:32.867230 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xd7gg" Sep 09 10:37:32.869726 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:32.869706 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 09 10:37:32.869816 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:32.869776 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-cjrrd\"" Sep 09 10:37:32.874076 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:32.874053 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-k45xt"] Sep 09 10:37:32.881027 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:32.880998 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-xd7gg"] Sep 09 10:37:33.012058 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:33.012016 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d352779-62ce-4e83-ac40-5de8a05026cc-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xd7gg\" (UID: \"3d352779-62ce-4e83-ac40-5de8a05026cc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xd7gg" Sep 09 10:37:33.012058 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:33.012058 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cfth\" (UniqueName: \"kubernetes.io/projected/2aa16cb0-c44e-48cd-986e-cc46caef5437-kube-api-access-6cfth\") pod \"kserve-controller-manager-d6f4b65b8-k45xt\" (UID: \"2aa16cb0-c44e-48cd-986e-cc46caef5437\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-k45xt" Sep 09 10:37:33.012281 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:33.012136 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2aa16cb0-c44e-48cd-986e-cc46caef5437-cert\") pod \"kserve-controller-manager-d6f4b65b8-k45xt\" (UID: \"2aa16cb0-c44e-48cd-986e-cc46caef5437\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-k45xt" Sep 09 10:37:33.012281 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:33.012204 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blmrg\" (UniqueName: \"kubernetes.io/projected/3d352779-62ce-4e83-ac40-5de8a05026cc-kube-api-access-blmrg\") pod \"llmisvc-controller-manager-777c6bd7d5-xd7gg\" (UID: \"3d352779-62ce-4e83-ac40-5de8a05026cc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xd7gg" Sep 09 10:37:33.113232 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:33.113145 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d352779-62ce-4e83-ac40-5de8a05026cc-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xd7gg\" (UID: \"3d352779-62ce-4e83-ac40-5de8a05026cc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xd7gg" Sep 09 10:37:33.113232 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:33.113191 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cfth\" (UniqueName: \"kubernetes.io/projected/2aa16cb0-c44e-48cd-986e-cc46caef5437-kube-api-access-6cfth\") pod \"kserve-controller-manager-d6f4b65b8-k45xt\" (UID: \"2aa16cb0-c44e-48cd-986e-cc46caef5437\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-k45xt" Sep 09 10:37:33.113232 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:33.113229 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2aa16cb0-c44e-48cd-986e-cc46caef5437-cert\") pod \"kserve-controller-manager-d6f4b65b8-k45xt\" (UID: \"2aa16cb0-c44e-48cd-986e-cc46caef5437\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-k45xt" Sep 09 10:37:33.113490 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:33.113264 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blmrg\" (UniqueName: \"kubernetes.io/projected/3d352779-62ce-4e83-ac40-5de8a05026cc-kube-api-access-blmrg\") pod \"llmisvc-controller-manager-777c6bd7d5-xd7gg\" (UID: \"3d352779-62ce-4e83-ac40-5de8a05026cc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xd7gg" Sep 09 10:37:33.113490 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:37:33.113395 2559 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Sep 09 10:37:33.113490 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:37:33.113470 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aa16cb0-c44e-48cd-986e-cc46caef5437-cert podName:2aa16cb0-c44e-48cd-986e-cc46caef5437 nodeName:}" failed. No retries permitted until 2026-09-09 10:37:33.613450468 +0000 UTC m=+693.788052785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2aa16cb0-c44e-48cd-986e-cc46caef5437-cert") pod "kserve-controller-manager-d6f4b65b8-k45xt" (UID: "2aa16cb0-c44e-48cd-986e-cc46caef5437") : secret "kserve-webhook-server-cert" not found Sep 09 10:37:33.115780 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:33.115748 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d352779-62ce-4e83-ac40-5de8a05026cc-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xd7gg\" (UID: \"3d352779-62ce-4e83-ac40-5de8a05026cc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xd7gg" Sep 09 10:37:33.147814 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:33.147786 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cfth\" (UniqueName: \"kubernetes.io/projected/2aa16cb0-c44e-48cd-986e-cc46caef5437-kube-api-access-6cfth\") pod \"kserve-controller-manager-d6f4b65b8-k45xt\" (UID: \"2aa16cb0-c44e-48cd-986e-cc46caef5437\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-k45xt" Sep 09 10:37:33.147814 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:33.147811 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blmrg\" (UniqueName: \"kubernetes.io/projected/3d352779-62ce-4e83-ac40-5de8a05026cc-kube-api-access-blmrg\") pod \"llmisvc-controller-manager-777c6bd7d5-xd7gg\" (UID: \"3d352779-62ce-4e83-ac40-5de8a05026cc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xd7gg" Sep 09 10:37:33.180843 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:33.180812 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xd7gg" Sep 09 10:37:33.474356 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:33.474279 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-xd7gg"] Sep 09 10:37:33.477121 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:37:33.477094 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3d352779_62ce_4e83_ac40_5de8a05026cc.slice/crio-7caebaaab8b8617bd8b29b1037ed3b6fce4afec2c8afd864c1ce8474d176b07a WatchSource:0}: Error finding container 7caebaaab8b8617bd8b29b1037ed3b6fce4afec2c8afd864c1ce8474d176b07a: Status 404 returned error can't find the container with id 7caebaaab8b8617bd8b29b1037ed3b6fce4afec2c8afd864c1ce8474d176b07a Sep 09 10:37:33.618797 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:33.618752 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2aa16cb0-c44e-48cd-986e-cc46caef5437-cert\") pod \"kserve-controller-manager-d6f4b65b8-k45xt\" (UID: \"2aa16cb0-c44e-48cd-986e-cc46caef5437\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-k45xt" Sep 09 10:37:33.621213 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:33.621188 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2aa16cb0-c44e-48cd-986e-cc46caef5437-cert\") pod \"kserve-controller-manager-d6f4b65b8-k45xt\" (UID: \"2aa16cb0-c44e-48cd-986e-cc46caef5437\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-k45xt" Sep 09 10:37:33.771751 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:33.771647 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-k45xt" Sep 09 10:37:33.797627 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:33.797599 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xd7gg" event={"ID":"3d352779-62ce-4e83-ac40-5de8a05026cc","Type":"ContainerStarted","Data":"7caebaaab8b8617bd8b29b1037ed3b6fce4afec2c8afd864c1ce8474d176b07a"} Sep 09 10:37:34.051368 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:34.051339 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-k45xt"] Sep 09 10:37:34.053751 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:37:34.053721 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aa16cb0_c44e_48cd_986e_cc46caef5437.slice/crio-04b6f3471fcd74a385c2a36498b8c75919688ebb258cae1bc837f22d752f4e89 WatchSource:0}: Error finding container 04b6f3471fcd74a385c2a36498b8c75919688ebb258cae1bc837f22d752f4e89: Status 404 returned error can't find the container with id 04b6f3471fcd74a385c2a36498b8c75919688ebb258cae1bc837f22d752f4e89 Sep 09 10:37:34.805857 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:34.805822 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-k45xt" event={"ID":"2aa16cb0-c44e-48cd-986e-cc46caef5437","Type":"ContainerStarted","Data":"04b6f3471fcd74a385c2a36498b8c75919688ebb258cae1bc837f22d752f4e89"} Sep 09 10:37:38.050800 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:38.050755 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 10:37:38.051092 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:38.050838 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 09 10:37:38.828692 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:38.828658 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-k45xt" event={"ID":"2aa16cb0-c44e-48cd-986e-cc46caef5437","Type":"ContainerStarted","Data":"8ef4757b41eb2b686a013aafa24ab95655c30fdd1dbd2f99eaf5a6dadee90201"} Sep 09 10:37:38.828919 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:38.828888 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-k45xt" Sep 09 10:37:38.830350 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:38.830321 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v2t7d" event={"ID":"af4f8698-6435-4c45-8a43-0d2eba16ce43","Type":"ContainerStarted","Data":"2367194317b6d5264a5387f39ed008335c1b3f7c15c3c7e82289d1be03791d48"} Sep 09 10:37:38.830475 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:38.830394 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v2t7d" Sep 09 10:37:38.831812 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:38.831790 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xd7gg" event={"ID":"3d352779-62ce-4e83-ac40-5de8a05026cc","Type":"ContainerStarted","Data":"d0a0ed3566a9dec1ffb11b087200c5769c275a5010ce2ca74934f93d4e4f7c2b"} Sep 09 10:37:38.831920 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:38.831909 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xd7gg" Sep 09 10:37:38.850698 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:38.850644 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-k45xt" podStartSLOduration=2.858112733 podStartE2EDuration="6.85062887s" totalImagesPullingTime="3.992516137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:37:38.848494527 +0000 UTC m=+699.023096877" watchObservedRunningTime="2026-09-09 10:37:38.85062887 +0000 UTC m=+699.025231194" Sep 09 10:37:38.868159 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:38.868107 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xd7gg" podStartSLOduration=2.292775506 podStartE2EDuration="6.868091893s" totalImagesPullingTime="4.575316387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:37:38.86711208 +0000 UTC m=+699.041714406" watchObservedRunningTime="2026-09-09 10:37:38.868091893 +0000 UTC m=+699.042694219" Sep 09 10:37:38.896047 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:38.895993 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v2t7d" podStartSLOduration=2.253515746 podStartE2EDuration="9.895979141s" totalImagesPullingTime="7.642463395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:37:38.894100713 +0000 UTC m=+699.068703037" watchObservedRunningTime="2026-09-09 10:37:38.895979141 +0000 UTC m=+699.070581466" Sep 09 10:37:39.836793 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:37:39.836766 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v2t7d" Sep 09 10:38:09.837671 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:38:09.837635 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xd7gg" Sep 09 10:38:09.840544 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:38:09.840489 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-k45xt" Sep 09 10:38:58.422879 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:38:58.422839 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-l6qc7"] Sep 09 10:38:58.425743 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:38:58.425726 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l6qc7" Sep 09 10:38:58.428493 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:38:58.428469 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-l429n\"" Sep 09 10:38:58.429352 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:38:58.429328 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 09 10:38:58.429447 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:38:58.429328 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 09 10:38:58.438662 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:38:58.438634 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-l6qc7"] Sep 09 10:38:58.501534 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:38:58.501473 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59tt5\" (UniqueName: \"kubernetes.io/projected/201bfbe0-9911-461c-8ede-1c4dbd3f2ddc-kube-api-access-59tt5\") pod \"limitador-operator-controller-manager-74495c69f8-l6qc7\" (UID: \"201bfbe0-9911-461c-8ede-1c4dbd3f2ddc\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l6qc7" Sep 09 10:38:58.602625 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:38:58.602584 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59tt5\" (UniqueName: \"kubernetes.io/projected/201bfbe0-9911-461c-8ede-1c4dbd3f2ddc-kube-api-access-59tt5\") pod \"limitador-operator-controller-manager-74495c69f8-l6qc7\" (UID: \"201bfbe0-9911-461c-8ede-1c4dbd3f2ddc\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l6qc7" Sep 09 10:38:58.612541 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:38:58.612492 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59tt5\" (UniqueName: \"kubernetes.io/projected/201bfbe0-9911-461c-8ede-1c4dbd3f2ddc-kube-api-access-59tt5\") pod \"limitador-operator-controller-manager-74495c69f8-l6qc7\" (UID: \"201bfbe0-9911-461c-8ede-1c4dbd3f2ddc\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l6qc7" Sep 09 10:38:58.736231 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:38:58.736136 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l6qc7" Sep 09 10:38:59.014140 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:38:59.014045 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-l6qc7"] Sep 09 10:38:59.183594 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:38:59.183544 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l6qc7" event={"ID":"201bfbe0-9911-461c-8ede-1c4dbd3f2ddc","Type":"ContainerStarted","Data":"e74dd81d77b2ef537abb47c9c72b2885abde9c16a12a5fae3c3fd6ce054315b2"} Sep 09 10:39:02.198984 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:02.198948 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l6qc7" event={"ID":"201bfbe0-9911-461c-8ede-1c4dbd3f2ddc","Type":"ContainerStarted","Data":"953d882bb95d68ef943215c313804adb09937d0ec78ae28730525e47bb07ed19"} Sep 09 10:39:02.199357 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:02.199087 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l6qc7" Sep 09 10:39:02.222590 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:02.222533 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l6qc7" podStartSLOduration=1.695111246 podStartE2EDuration="4.222515095s" totalImagesPullingTime="2.527403849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:38:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:39:02.219384141 +0000 UTC m=+782.393986465" watchObservedRunningTime="2026-09-09 10:39:02.222515095 +0000 UTC m=+782.397117417" Sep 09 10:39:13.205859 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:13.205823 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l6qc7" Sep 09 10:39:15.915664 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:15.915630 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-l6qc7"] Sep 09 10:39:15.916074 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:15.915838 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l6qc7" podUID="201bfbe0-9911-461c-8ede-1c4dbd3f2ddc" containerName="manager" containerID="cri-o://953d882bb95d68ef943215c313804adb09937d0ec78ae28730525e47bb07ed19" gracePeriod=2 Sep 09 10:39:15.926841 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:15.926808 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-l6qc7"] Sep 09 10:39:15.943979 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:15.943947 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-g2tnk"] Sep 09 10:39:15.944257 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:15.944245 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="201bfbe0-9911-461c-8ede-1c4dbd3f2ddc" containerName="manager" Sep 09 10:39:15.944298 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:15.944259 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="201bfbe0-9911-461c-8ede-1c4dbd3f2ddc" containerName="manager" Sep 09 10:39:15.944331 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:15.944308 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="201bfbe0-9911-461c-8ede-1c4dbd3f2ddc" containerName="manager" Sep 09 10:39:15.948104 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:15.948074 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g2tnk" Sep 09 10:39:15.953101 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:39:15.953058 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-l6qc7\" is forbidden: User \"system:node:ip-10-0-139-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-133.ec2.internal' and this object" podUID="201bfbe0-9911-461c-8ede-1c4dbd3f2ddc" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l6qc7" Sep 09 10:39:15.963062 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:15.963029 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-g2tnk"] Sep 09 10:39:16.024725 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:16.024694 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn96m\" (UniqueName: \"kubernetes.io/projected/2e5f3a17-b2c5-4e9e-bca2-00c821c9b0a6-kube-api-access-mn96m\") pod \"limitador-operator-controller-manager-74495c69f8-g2tnk\" (UID: \"2e5f3a17-b2c5-4e9e-bca2-00c821c9b0a6\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g2tnk" Sep 09 10:39:16.125902 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:16.125864 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn96m\" (UniqueName: \"kubernetes.io/projected/2e5f3a17-b2c5-4e9e-bca2-00c821c9b0a6-kube-api-access-mn96m\") pod \"limitador-operator-controller-manager-74495c69f8-g2tnk\" (UID: \"2e5f3a17-b2c5-4e9e-bca2-00c821c9b0a6\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g2tnk" Sep 09 10:39:16.141823 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:16.141801 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn96m\" (UniqueName: \"kubernetes.io/projected/2e5f3a17-b2c5-4e9e-bca2-00c821c9b0a6-kube-api-access-mn96m\") pod \"limitador-operator-controller-manager-74495c69f8-g2tnk\" (UID: \"2e5f3a17-b2c5-4e9e-bca2-00c821c9b0a6\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g2tnk" Sep 09 10:39:16.157479 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:16.157448 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l6qc7" Sep 09 10:39:16.160462 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:39:16.160436 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-l6qc7\" is forbidden: User \"system:node:ip-10-0-139-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-133.ec2.internal' and this object" podUID="201bfbe0-9911-461c-8ede-1c4dbd3f2ddc" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l6qc7" Sep 09 10:39:16.227133 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:16.227049 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59tt5\" (UniqueName: \"kubernetes.io/projected/201bfbe0-9911-461c-8ede-1c4dbd3f2ddc-kube-api-access-59tt5\") pod \"201bfbe0-9911-461c-8ede-1c4dbd3f2ddc\" (UID: \"201bfbe0-9911-461c-8ede-1c4dbd3f2ddc\") " Sep 09 10:39:16.229266 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:16.229238 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/201bfbe0-9911-461c-8ede-1c4dbd3f2ddc-kube-api-access-59tt5" (OuterVolumeSpecName: "kube-api-access-59tt5") pod "201bfbe0-9911-461c-8ede-1c4dbd3f2ddc" (UID: "201bfbe0-9911-461c-8ede-1c4dbd3f2ddc"). InnerVolumeSpecName "kube-api-access-59tt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:39:16.258813 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:16.258782 2559 generic.go:356] "Generic (PLEG): container finished" podID="201bfbe0-9911-461c-8ede-1c4dbd3f2ddc" containerID="953d882bb95d68ef943215c313804adb09937d0ec78ae28730525e47bb07ed19" exitCode=0 Sep 09 10:39:16.259001 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:16.258825 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l6qc7" Sep 09 10:39:16.259001 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:16.258886 2559 scope.go:117] "RemoveContainer" containerID="953d882bb95d68ef943215c313804adb09937d0ec78ae28730525e47bb07ed19" Sep 09 10:39:16.261410 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:39:16.261380 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-l6qc7\" is forbidden: User \"system:node:ip-10-0-139-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-133.ec2.internal' and this object" podUID="201bfbe0-9911-461c-8ede-1c4dbd3f2ddc" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l6qc7" Sep 09 10:39:16.267495 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:16.267477 2559 scope.go:117] "RemoveContainer" containerID="953d882bb95d68ef943215c313804adb09937d0ec78ae28730525e47bb07ed19" Sep 09 10:39:16.267790 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:39:16.267770 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"953d882bb95d68ef943215c313804adb09937d0ec78ae28730525e47bb07ed19\": container with ID starting with 953d882bb95d68ef943215c313804adb09937d0ec78ae28730525e47bb07ed19 not found: ID does not exist" containerID="953d882bb95d68ef943215c313804adb09937d0ec78ae28730525e47bb07ed19" Sep 09 10:39:16.267871 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:16.267802 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"953d882bb95d68ef943215c313804adb09937d0ec78ae28730525e47bb07ed19"} err="failed to get container status \"953d882bb95d68ef943215c313804adb09937d0ec78ae28730525e47bb07ed19\": rpc error: code = NotFound desc = could not find container \"953d882bb95d68ef943215c313804adb09937d0ec78ae28730525e47bb07ed19\": container with ID starting with 953d882bb95d68ef943215c313804adb09937d0ec78ae28730525e47bb07ed19 not found: ID does not exist" Sep 09 10:39:16.270392 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:39:16.270365 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-l6qc7\" is forbidden: User \"system:node:ip-10-0-139-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-133.ec2.internal' and this object" podUID="201bfbe0-9911-461c-8ede-1c4dbd3f2ddc" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l6qc7" Sep 09 10:39:16.306981 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:16.306943 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g2tnk" Sep 09 10:39:16.328006 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:16.327973 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59tt5\" (UniqueName: \"kubernetes.io/projected/201bfbe0-9911-461c-8ede-1c4dbd3f2ddc-kube-api-access-59tt5\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:39:16.334066 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:16.334031 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="201bfbe0-9911-461c-8ede-1c4dbd3f2ddc" path="/var/lib/kubelet/pods/201bfbe0-9911-461c-8ede-1c4dbd3f2ddc/volumes" Sep 09 10:39:16.584952 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:16.584920 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-g2tnk"] Sep 09 10:39:16.588558 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:39:16.588529 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e5f3a17_b2c5_4e9e_bca2_00c821c9b0a6.slice/crio-fbd5a3b4b51fe14473d4a532577d13c3baad8126c1c82882a3c712cffc7d4f7e WatchSource:0}: Error finding container fbd5a3b4b51fe14473d4a532577d13c3baad8126c1c82882a3c712cffc7d4f7e: Status 404 returned error can't find the container with id fbd5a3b4b51fe14473d4a532577d13c3baad8126c1c82882a3c712cffc7d4f7e Sep 09 10:39:17.265593 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:17.265558 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g2tnk" event={"ID":"2e5f3a17-b2c5-4e9e-bca2-00c821c9b0a6","Type":"ContainerStarted","Data":"61a54c79d1be16daf80b1fdf73ce1f59c2f255fecdea7c3db54e37aeff17b759"} Sep 09 10:39:17.266013 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:17.265609 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g2tnk" event={"ID":"2e5f3a17-b2c5-4e9e-bca2-00c821c9b0a6","Type":"ContainerStarted","Data":"fbd5a3b4b51fe14473d4a532577d13c3baad8126c1c82882a3c712cffc7d4f7e"} Sep 09 10:39:17.266013 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:17.265632 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g2tnk" Sep 09 10:39:17.288704 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:17.288653 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g2tnk" podStartSLOduration=2.288638649 podStartE2EDuration="2.288638649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:39:17.286782638 +0000 UTC m=+797.461384988" watchObservedRunningTime="2026-09-09 10:39:17.288638649 +0000 UTC m=+797.463240974" Sep 09 10:39:27.983232 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:27.983157 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hk5f5"] Sep 09 10:39:27.986232 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:27.986216 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hk5f5" Sep 09 10:39:27.988675 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:39:27.988640 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-hk5f5\" is forbidden: User \"system:node:ip-10-0-139-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-133.ec2.internal' and this object" podUID="9ae40ea5-d767-4a0d-a27b-5a246d66a83c" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hk5f5" Sep 09 10:39:27.988804 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:39:27.988695 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kuadrant-operator-controller-manager-dockercfg-x9wq2\" is forbidden: User \"system:node:ip-10-0-139-133.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-133.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-x9wq2\"" type="*v1.Secret" Sep 09 10:39:28.005011 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:28.004976 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hk5f5"] Sep 09 10:39:28.017722 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:28.017693 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t74p\" (UniqueName: \"kubernetes.io/projected/9ae40ea5-d767-4a0d-a27b-5a246d66a83c-kube-api-access-2t74p\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hk5f5\" (UID: \"9ae40ea5-d767-4a0d-a27b-5a246d66a83c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hk5f5" Sep 09 10:39:28.017848 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:28.017728 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ae40ea5-d767-4a0d-a27b-5a246d66a83c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hk5f5\" (UID: \"9ae40ea5-d767-4a0d-a27b-5a246d66a83c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hk5f5" Sep 09 10:39:28.118754 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:28.118715 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t74p\" (UniqueName: \"kubernetes.io/projected/9ae40ea5-d767-4a0d-a27b-5a246d66a83c-kube-api-access-2t74p\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hk5f5\" (UID: \"9ae40ea5-d767-4a0d-a27b-5a246d66a83c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hk5f5" Sep 09 10:39:28.118754 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:28.118754 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ae40ea5-d767-4a0d-a27b-5a246d66a83c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hk5f5\" (UID: \"9ae40ea5-d767-4a0d-a27b-5a246d66a83c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hk5f5" Sep 09 10:39:28.119098 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:28.119081 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ae40ea5-d767-4a0d-a27b-5a246d66a83c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hk5f5\" (UID: \"9ae40ea5-d767-4a0d-a27b-5a246d66a83c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hk5f5" Sep 09 10:39:28.132536 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:28.132489 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t74p\" (UniqueName: \"kubernetes.io/projected/9ae40ea5-d767-4a0d-a27b-5a246d66a83c-kube-api-access-2t74p\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hk5f5\" (UID: \"9ae40ea5-d767-4a0d-a27b-5a246d66a83c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hk5f5" Sep 09 10:39:28.273695 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:28.273611 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g2tnk" Sep 09 10:39:29.297586 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:29.297545 2559 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hk5f5" secret="" err="failed to sync secret cache: timed out waiting for the condition" Sep 09 10:39:29.298075 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:29.297622 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hk5f5" Sep 09 10:39:29.509702 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:29.509673 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-x9wq2\"" Sep 09 10:39:29.585274 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:29.585238 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hk5f5"] Sep 09 10:39:29.588691 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:39:29.588667 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae40ea5_d767_4a0d_a27b_5a246d66a83c.slice/crio-b867ae70df78f97a3432f35e53a2423a3b7833306a5e140fcefd0bf446730dbc WatchSource:0}: Error finding container b867ae70df78f97a3432f35e53a2423a3b7833306a5e140fcefd0bf446730dbc: Status 404 returned error can't find the container with id b867ae70df78f97a3432f35e53a2423a3b7833306a5e140fcefd0bf446730dbc Sep 09 10:39:30.324637 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:30.324600 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hk5f5" event={"ID":"9ae40ea5-d767-4a0d-a27b-5a246d66a83c","Type":"ContainerStarted","Data":"b867ae70df78f97a3432f35e53a2423a3b7833306a5e140fcefd0bf446730dbc"} Sep 09 10:39:33.340577 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:33.340536 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hk5f5" event={"ID":"9ae40ea5-d767-4a0d-a27b-5a246d66a83c","Type":"ContainerStarted","Data":"856b9208c2571b532035938dc396c62bb56dcf3a55e7c351b0c33bee815ac50f"} Sep 09 10:39:33.340997 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:33.340750 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hk5f5" Sep 09 10:39:33.365572 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:33.365487 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hk5f5" podStartSLOduration=2.937304132 podStartE2EDuration="6.365469699s" totalImagesPullingTime="3.428165567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:39:33.363155178 +0000 UTC m=+813.537757503" watchObservedRunningTime="2026-09-09 10:39:33.365469699 +0000 UTC m=+813.540072024" Sep 09 10:39:44.347760 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:39:44.347724 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hk5f5" Sep 09 10:41:00.270641 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:41:00.270613 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d22ff_0a56b285-3f3c-4f6d-a939-8a95ae00e8da/ovn-acl-logging/0.log" Sep 09 10:41:00.272401 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:41:00.272377 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d22ff_0a56b285-3f3c-4f6d-a939-8a95ae00e8da/ovn-acl-logging/0.log" Sep 09 10:41:19.876566 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:41:19.876528 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-x4njj"] Sep 09 10:41:19.885816 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:41:19.885781 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-x4njj" Sep 09 10:41:19.888485 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:41:19.888459 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 09 10:41:19.888639 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:41:19.888490 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-lspjv\"" Sep 09 10:41:19.895043 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:41:19.894968 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-x4njj"] Sep 09 10:41:20.029321 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:41:20.029287 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cdfe780-23e6-4fa2-bb10-dd57f4c5c25b-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-x4njj\" (UID: \"1cdfe780-23e6-4fa2-bb10-dd57f4c5c25b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-x4njj" Sep 09 10:41:20.029321 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:41:20.029324 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rkxl\" (UniqueName: \"kubernetes.io/projected/1cdfe780-23e6-4fa2-bb10-dd57f4c5c25b-kube-api-access-8rkxl\") pod \"payload-processing-5b54b7fbb7-x4njj\" (UID: \"1cdfe780-23e6-4fa2-bb10-dd57f4c5c25b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-x4njj" Sep 09 10:41:20.130705 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:41:20.130608 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cdfe780-23e6-4fa2-bb10-dd57f4c5c25b-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-x4njj\" (UID: \"1cdfe780-23e6-4fa2-bb10-dd57f4c5c25b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-x4njj" Sep 09 10:41:20.130705 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:41:20.130650 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rkxl\" (UniqueName: \"kubernetes.io/projected/1cdfe780-23e6-4fa2-bb10-dd57f4c5c25b-kube-api-access-8rkxl\") pod \"payload-processing-5b54b7fbb7-x4njj\" (UID: \"1cdfe780-23e6-4fa2-bb10-dd57f4c5c25b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-x4njj" Sep 09 10:41:20.131487 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:41:20.131432 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cdfe780-23e6-4fa2-bb10-dd57f4c5c25b-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-x4njj\" (UID: \"1cdfe780-23e6-4fa2-bb10-dd57f4c5c25b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-x4njj" Sep 09 10:41:20.147273 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:41:20.147241 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rkxl\" (UniqueName: \"kubernetes.io/projected/1cdfe780-23e6-4fa2-bb10-dd57f4c5c25b-kube-api-access-8rkxl\") pod \"payload-processing-5b54b7fbb7-x4njj\" (UID: \"1cdfe780-23e6-4fa2-bb10-dd57f4c5c25b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-x4njj" Sep 09 10:41:20.200081 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:41:20.200034 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-x4njj" Sep 09 10:41:20.478851 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:41:20.478764 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-x4njj"] Sep 09 10:41:20.481876 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:41:20.481837 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cdfe780_23e6_4fa2_bb10_dd57f4c5c25b.slice/crio-6c0c9e986a2670411c5778258ee1726bda1da932d7bf6f935e23620e767442da WatchSource:0}: Error finding container 6c0c9e986a2670411c5778258ee1726bda1da932d7bf6f935e23620e767442da: Status 404 returned error can't find the container with id 6c0c9e986a2670411c5778258ee1726bda1da932d7bf6f935e23620e767442da Sep 09 10:41:20.809703 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:41:20.809617 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-x4njj" event={"ID":"1cdfe780-23e6-4fa2-bb10-dd57f4c5c25b","Type":"ContainerStarted","Data":"6c0c9e986a2670411c5778258ee1726bda1da932d7bf6f935e23620e767442da"} Sep 09 10:41:22.825662 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:41:22.825629 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-x4njj" event={"ID":"1cdfe780-23e6-4fa2-bb10-dd57f4c5c25b","Type":"ContainerStarted","Data":"cec506ebbda02750f7b8db58a50e9e75013da37769d43e36ec4367a2150afcf8"} Sep 09 10:41:22.826032 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:41:22.825720 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-x4njj" Sep 09 10:41:22.845535 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:41:22.845462 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-x4njj" podStartSLOduration=1.999164328 podStartE2EDuration="3.845445006s" totalImagesPullingTime="1.846280678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:41:22.843852658 +0000 UTC m=+923.018454974" watchObservedRunningTime="2026-09-09 10:41:22.845445006 +0000 UTC m=+923.020047331" Sep 09 10:41:33.832730 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:41:33.832696 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-x4njj" Sep 09 10:42:08.207049 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:08.207014 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-k26kt"] Sep 09 10:42:08.210366 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:08.210346 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k26kt" Sep 09 10:42:08.213079 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:08.213042 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 09 10:42:08.213193 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:08.213077 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 09 10:42:08.213193 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:08.213052 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 09 10:42:08.214206 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:08.214189 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2mjpr\"" Sep 09 10:42:08.221536 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:08.221478 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-k26kt"] Sep 09 10:42:08.328439 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:08.328400 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k26kt\" (UID: \"b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k26kt" Sep 09 10:42:08.328439 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:08.328436 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k26kt\" (UID: \"b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k26kt" Sep 09 10:42:08.328700 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:08.328453 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcvpc\" (UniqueName: \"kubernetes.io/projected/b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db-kube-api-access-vcvpc\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k26kt\" (UID: \"b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k26kt" Sep 09 10:42:08.328882 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:08.328861 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k26kt\" (UID: \"b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k26kt" Sep 09 10:42:08.328952 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:08.328930 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k26kt\" (UID: \"b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k26kt" Sep 09 10:42:08.329094 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:08.329066 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k26kt\" (UID: \"b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k26kt" Sep 09 10:42:08.429634 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:08.429582 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k26kt\" (UID: \"b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k26kt" Sep 09 10:42:08.429634 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:08.429632 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k26kt\" (UID: \"b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k26kt" Sep 09 10:42:08.429878 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:08.429659 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcvpc\" (UniqueName: \"kubernetes.io/projected/b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db-kube-api-access-vcvpc\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k26kt\" (UID: \"b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k26kt" Sep 09 10:42:08.429878 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:08.429679 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k26kt\" (UID: \"b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k26kt" Sep 09 10:42:08.429878 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:08.429707 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k26kt\" (UID: \"b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k26kt" Sep 09 10:42:08.429878 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:08.429756 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k26kt\" (UID: \"b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k26kt" Sep 09 10:42:08.430084 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:08.430062 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k26kt\" (UID: \"b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k26kt" Sep 09 10:42:08.430084 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:08.430070 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k26kt\" (UID: \"b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k26kt" Sep 09 10:42:08.430175 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:08.430154 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k26kt\" (UID: \"b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k26kt" Sep 09 10:42:08.432046 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:08.432021 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k26kt\" (UID: \"b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k26kt" Sep 09 10:42:08.432277 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:08.432261 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k26kt\" (UID: \"b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k26kt" Sep 09 10:42:08.439540 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:08.439518 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcvpc\" (UniqueName: \"kubernetes.io/projected/b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db-kube-api-access-vcvpc\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k26kt\" (UID: \"b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k26kt" Sep 09 10:42:08.521079 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:08.520986 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k26kt" Sep 09 10:42:08.798186 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:08.798115 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-k26kt"] Sep 09 10:42:08.801482 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:42:08.801458 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb38da3c8_0e6e_4ffc_b5ec_0d8d8f1ab1db.slice/crio-e7e1d302e18b0bba09b823612bf0f3d0fb8688c4d24549d26f6914eb389e162a WatchSource:0}: Error finding container e7e1d302e18b0bba09b823612bf0f3d0fb8688c4d24549d26f6914eb389e162a: Status 404 returned error can't find the container with id e7e1d302e18b0bba09b823612bf0f3d0fb8688c4d24549d26f6914eb389e162a Sep 09 10:42:09.038812 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:09.038768 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k26kt" event={"ID":"b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db","Type":"ContainerStarted","Data":"e7e1d302e18b0bba09b823612bf0f3d0fb8688c4d24549d26f6914eb389e162a"} Sep 09 10:42:10.046254 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:10.046210 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k26kt" event={"ID":"b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db","Type":"ContainerStarted","Data":"1ec95c60218ca538b27a764e1cf5495d13cba39cac5dd40352ddf0f1ac682cd8"} Sep 09 10:42:10.066554 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:10.066409 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k26kt" podStartSLOduration=1.055450782 podStartE2EDuration="2.066389131s" totalImagesPullingTime="1.010938349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:42:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:42:10.064979919 +0000 UTC m=+970.239582244" watchObservedRunningTime="2026-09-09 10:42:10.066389131 +0000 UTC m=+970.240991457" Sep 09 10:42:18.521957 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:18.521915 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k26kt" Sep 09 10:42:18.522446 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:18.521991 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k26kt" Sep 09 10:42:18.534985 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:18.534954 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k26kt" Sep 09 10:42:19.099377 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:19.099343 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k26kt" Sep 09 10:42:46.609626 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:46.609537 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zpsd9"] Sep 09 10:42:46.612060 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:46.612038 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zpsd9" Sep 09 10:42:46.614831 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:46.614809 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 09 10:42:46.623677 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:46.623651 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zpsd9"] Sep 09 10:42:46.638994 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:46.638959 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d9be21b-3c67-4186-903f-8d6a7e76038c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zpsd9\" (UID: \"2d9be21b-3c67-4186-903f-8d6a7e76038c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zpsd9" Sep 09 10:42:46.639169 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:46.639005 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d9be21b-3c67-4186-903f-8d6a7e76038c-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zpsd9\" (UID: \"2d9be21b-3c67-4186-903f-8d6a7e76038c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zpsd9" Sep 09 10:42:46.639169 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:46.639074 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmtdj\" (UniqueName: \"kubernetes.io/projected/2d9be21b-3c67-4186-903f-8d6a7e76038c-kube-api-access-jmtdj\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zpsd9\" (UID: \"2d9be21b-3c67-4186-903f-8d6a7e76038c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zpsd9" Sep 09 10:42:46.639169 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:46.639104 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d9be21b-3c67-4186-903f-8d6a7e76038c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zpsd9\" (UID: \"2d9be21b-3c67-4186-903f-8d6a7e76038c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zpsd9" Sep 09 10:42:46.639169 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:46.639144 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d9be21b-3c67-4186-903f-8d6a7e76038c-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zpsd9\" (UID: \"2d9be21b-3c67-4186-903f-8d6a7e76038c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zpsd9" Sep 09 10:42:46.639169 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:46.639164 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d9be21b-3c67-4186-903f-8d6a7e76038c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zpsd9\" (UID: \"2d9be21b-3c67-4186-903f-8d6a7e76038c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zpsd9" Sep 09 10:42:46.740415 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:46.740376 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d9be21b-3c67-4186-903f-8d6a7e76038c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zpsd9\" (UID: \"2d9be21b-3c67-4186-903f-8d6a7e76038c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zpsd9" Sep 09 10:42:46.740625 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:46.740437 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d9be21b-3c67-4186-903f-8d6a7e76038c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zpsd9\" (UID: \"2d9be21b-3c67-4186-903f-8d6a7e76038c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zpsd9" Sep 09 10:42:46.740625 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:46.740484 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d9be21b-3c67-4186-903f-8d6a7e76038c-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zpsd9\" (UID: \"2d9be21b-3c67-4186-903f-8d6a7e76038c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zpsd9" Sep 09 10:42:46.740625 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:46.740547 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmtdj\" (UniqueName: \"kubernetes.io/projected/2d9be21b-3c67-4186-903f-8d6a7e76038c-kube-api-access-jmtdj\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zpsd9\" (UID: \"2d9be21b-3c67-4186-903f-8d6a7e76038c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zpsd9" Sep 09 10:42:46.740625 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:46.740575 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d9be21b-3c67-4186-903f-8d6a7e76038c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zpsd9\" (UID: \"2d9be21b-3c67-4186-903f-8d6a7e76038c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zpsd9" Sep 09 10:42:46.740625 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:46.740619 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d9be21b-3c67-4186-903f-8d6a7e76038c-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zpsd9\" (UID: \"2d9be21b-3c67-4186-903f-8d6a7e76038c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zpsd9" Sep 09 10:42:46.740887 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:46.740862 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d9be21b-3c67-4186-903f-8d6a7e76038c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zpsd9\" (UID: \"2d9be21b-3c67-4186-903f-8d6a7e76038c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zpsd9" Sep 09 10:42:46.740976 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:46.740908 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d9be21b-3c67-4186-903f-8d6a7e76038c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zpsd9\" (UID: \"2d9be21b-3c67-4186-903f-8d6a7e76038c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zpsd9" Sep 09 10:42:46.740976 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:46.740951 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d9be21b-3c67-4186-903f-8d6a7e76038c-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zpsd9\" (UID: \"2d9be21b-3c67-4186-903f-8d6a7e76038c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zpsd9" Sep 09 10:42:46.742925 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:46.742891 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d9be21b-3c67-4186-903f-8d6a7e76038c-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zpsd9\" (UID: \"2d9be21b-3c67-4186-903f-8d6a7e76038c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zpsd9" Sep 09 10:42:46.743267 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:46.743249 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d9be21b-3c67-4186-903f-8d6a7e76038c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zpsd9\" (UID: \"2d9be21b-3c67-4186-903f-8d6a7e76038c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zpsd9" Sep 09 10:42:46.749901 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:46.749874 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmtdj\" (UniqueName: \"kubernetes.io/projected/2d9be21b-3c67-4186-903f-8d6a7e76038c-kube-api-access-jmtdj\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zpsd9\" (UID: \"2d9be21b-3c67-4186-903f-8d6a7e76038c\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zpsd9" Sep 09 10:42:46.922885 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:46.922799 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zpsd9" Sep 09 10:42:47.196837 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:47.196760 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zpsd9"] Sep 09 10:42:47.200643 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:42:47.200609 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d9be21b_3c67_4186_903f_8d6a7e76038c.slice/crio-143328d0224527438b1fe01508dd1cdcf4f32eb5609804aa7ad767be0deb5d94 WatchSource:0}: Error finding container 143328d0224527438b1fe01508dd1cdcf4f32eb5609804aa7ad767be0deb5d94: Status 404 returned error can't find the container with id 143328d0224527438b1fe01508dd1cdcf4f32eb5609804aa7ad767be0deb5d94 Sep 09 10:42:47.201982 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:47.201965 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 10:42:47.217371 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:47.217347 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zpsd9" event={"ID":"2d9be21b-3c67-4186-903f-8d6a7e76038c","Type":"ContainerStarted","Data":"143328d0224527438b1fe01508dd1cdcf4f32eb5609804aa7ad767be0deb5d94"} Sep 09 10:42:48.223681 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:48.223647 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zpsd9" event={"ID":"2d9be21b-3c67-4186-903f-8d6a7e76038c","Type":"ContainerStarted","Data":"d65c96d4c5a35c60c1decb750ccffb891b922dee6e4e823189f58ad2ee5dc6d0"} Sep 09 10:42:48.250225 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:48.250164 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zpsd9" podStartSLOduration=2.043976903 podStartE2EDuration="2.250146424s" totalImagesPullingTime="206.169521ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:42:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:42:48.248042971 +0000 UTC m=+1008.422645297" watchObservedRunningTime="2026-09-09 10:42:48.250146424 +0000 UTC m=+1008.424748748" Sep 09 10:42:56.923183 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:56.923127 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zpsd9" Sep 09 10:42:56.923808 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:56.923220 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zpsd9" Sep 09 10:42:56.935716 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:56.935687 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zpsd9" Sep 09 10:42:57.280119 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:42:57.280046 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zpsd9" Sep 09 10:44:43.250850 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:43.250813 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-40f1e9-5d4c4cf546-qvvtb"] Sep 09 10:44:43.254207 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:43.254186 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40f1e9-5d4c4cf546-qvvtb" Sep 09 10:44:43.256794 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:43.256772 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-40f1e9\"" Sep 09 10:44:43.263450 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:43.263423 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-40f1e9-5d4c4cf546-qvvtb"] Sep 09 10:44:43.280366 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:43.280332 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-40f1e9-56bfcc87ff-9j7vl"] Sep 09 10:44:43.284043 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:43.284020 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-40f1e9-56bfcc87ff-9j7vl" Sep 09 10:44:43.286753 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:43.286729 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-40f1e9-dockercfg-q927l\"" Sep 09 10:44:43.298402 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:43.298375 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-40f1e9-56bfcc87ff-9j7vl"] Sep 09 10:44:43.323189 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:43.323156 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcgmr\" (UniqueName: \"kubernetes.io/projected/b698c646-6e14-4275-bc2f-3d731843a697-kube-api-access-qcgmr\") pod \"payload-pre-processing-e2e-inf-a-40f1e9-5d4c4cf546-qvvtb\" (UID: \"b698c646-6e14-4275-bc2f-3d731843a697\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40f1e9-5d4c4cf546-qvvtb" Sep 09 10:44:43.323189 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:43.323191 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b698c646-6e14-4275-bc2f-3d731843a697-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-40f1e9-5d4c4cf546-qvvtb\" (UID: \"b698c646-6e14-4275-bc2f-3d731843a697\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40f1e9-5d4c4cf546-qvvtb" Sep 09 10:44:43.323401 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:43.323211 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffc08e84-a6f3-4d36-8969-c83a498924aa-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-40f1e9-56bfcc87ff-9j7vl\" (UID: \"ffc08e84-a6f3-4d36-8969-c83a498924aa\") " pod="openshift-ingress/payload-processing-e2e-inf-a-40f1e9-56bfcc87ff-9j7vl" Sep 09 10:44:43.323401 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:43.323227 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t66vm\" (UniqueName: \"kubernetes.io/projected/ffc08e84-a6f3-4d36-8969-c83a498924aa-kube-api-access-t66vm\") pod \"payload-processing-e2e-inf-a-40f1e9-56bfcc87ff-9j7vl\" (UID: \"ffc08e84-a6f3-4d36-8969-c83a498924aa\") " pod="openshift-ingress/payload-processing-e2e-inf-a-40f1e9-56bfcc87ff-9j7vl" Sep 09 10:44:43.424726 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:43.424683 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcgmr\" (UniqueName: \"kubernetes.io/projected/b698c646-6e14-4275-bc2f-3d731843a697-kube-api-access-qcgmr\") pod \"payload-pre-processing-e2e-inf-a-40f1e9-5d4c4cf546-qvvtb\" (UID: \"b698c646-6e14-4275-bc2f-3d731843a697\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40f1e9-5d4c4cf546-qvvtb" Sep 09 10:44:43.424726 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:43.424731 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b698c646-6e14-4275-bc2f-3d731843a697-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-40f1e9-5d4c4cf546-qvvtb\" (UID: \"b698c646-6e14-4275-bc2f-3d731843a697\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40f1e9-5d4c4cf546-qvvtb" Sep 09 10:44:43.424990 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:43.424749 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffc08e84-a6f3-4d36-8969-c83a498924aa-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-40f1e9-56bfcc87ff-9j7vl\" (UID: \"ffc08e84-a6f3-4d36-8969-c83a498924aa\") " pod="openshift-ingress/payload-processing-e2e-inf-a-40f1e9-56bfcc87ff-9j7vl" Sep 09 10:44:43.424990 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:43.424765 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t66vm\" (UniqueName: \"kubernetes.io/projected/ffc08e84-a6f3-4d36-8969-c83a498924aa-kube-api-access-t66vm\") pod \"payload-processing-e2e-inf-a-40f1e9-56bfcc87ff-9j7vl\" (UID: \"ffc08e84-a6f3-4d36-8969-c83a498924aa\") " pod="openshift-ingress/payload-processing-e2e-inf-a-40f1e9-56bfcc87ff-9j7vl" Sep 09 10:44:43.425519 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:43.425483 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b698c646-6e14-4275-bc2f-3d731843a697-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-40f1e9-5d4c4cf546-qvvtb\" (UID: \"b698c646-6e14-4275-bc2f-3d731843a697\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40f1e9-5d4c4cf546-qvvtb" Sep 09 10:44:43.425620 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:43.425483 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffc08e84-a6f3-4d36-8969-c83a498924aa-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-40f1e9-56bfcc87ff-9j7vl\" (UID: \"ffc08e84-a6f3-4d36-8969-c83a498924aa\") " pod="openshift-ingress/payload-processing-e2e-inf-a-40f1e9-56bfcc87ff-9j7vl" Sep 09 10:44:43.432589 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:43.432568 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcgmr\" (UniqueName: \"kubernetes.io/projected/b698c646-6e14-4275-bc2f-3d731843a697-kube-api-access-qcgmr\") pod \"payload-pre-processing-e2e-inf-a-40f1e9-5d4c4cf546-qvvtb\" (UID: \"b698c646-6e14-4275-bc2f-3d731843a697\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40f1e9-5d4c4cf546-qvvtb" Sep 09 10:44:43.432589 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:43.432588 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t66vm\" (UniqueName: \"kubernetes.io/projected/ffc08e84-a6f3-4d36-8969-c83a498924aa-kube-api-access-t66vm\") pod \"payload-processing-e2e-inf-a-40f1e9-56bfcc87ff-9j7vl\" (UID: \"ffc08e84-a6f3-4d36-8969-c83a498924aa\") " pod="openshift-ingress/payload-processing-e2e-inf-a-40f1e9-56bfcc87ff-9j7vl" Sep 09 10:44:43.564912 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:43.564824 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40f1e9-5d4c4cf546-qvvtb" Sep 09 10:44:43.594289 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:43.594258 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-40f1e9-56bfcc87ff-9j7vl" Sep 09 10:44:43.842284 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:43.842243 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-40f1e9-5d4c4cf546-qvvtb"] Sep 09 10:44:43.845399 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:44:43.845365 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb698c646_6e14_4275_bc2f_3d731843a697.slice/crio-d9b6fb0610db5193f2eecaf1f52024b6d5967cd7477c1c47aeb6d8d9b0fd7418 WatchSource:0}: Error finding container d9b6fb0610db5193f2eecaf1f52024b6d5967cd7477c1c47aeb6d8d9b0fd7418: Status 404 returned error can't find the container with id d9b6fb0610db5193f2eecaf1f52024b6d5967cd7477c1c47aeb6d8d9b0fd7418 Sep 09 10:44:43.883718 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:43.883695 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-40f1e9-56bfcc87ff-9j7vl"] Sep 09 10:44:43.889574 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:44:43.889549 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc08e84_a6f3_4d36_8969_c83a498924aa.slice/crio-896f9600c61ea7e3c7836a049ec2a31e33d4a448174fc637421f70bad8bb2a01 WatchSource:0}: Error finding container 896f9600c61ea7e3c7836a049ec2a31e33d4a448174fc637421f70bad8bb2a01: Status 404 returned error can't find the container with id 896f9600c61ea7e3c7836a049ec2a31e33d4a448174fc637421f70bad8bb2a01 Sep 09 10:44:44.764213 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:44.764173 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40f1e9-5d4c4cf546-qvvtb" event={"ID":"b698c646-6e14-4275-bc2f-3d731843a697","Type":"ContainerStarted","Data":"32c6f4de2297096d3c6afd84d29f3b55968706bf1e276ae1a5f72f313b66faed"} Sep 09 10:44:44.764775 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:44.764746 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40f1e9-5d4c4cf546-qvvtb" event={"ID":"b698c646-6e14-4275-bc2f-3d731843a697","Type":"ContainerStarted","Data":"d9b6fb0610db5193f2eecaf1f52024b6d5967cd7477c1c47aeb6d8d9b0fd7418"} Sep 09 10:44:44.764908 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:44.764792 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40f1e9-5d4c4cf546-qvvtb" Sep 09 10:44:44.765820 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:44.765794 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-40f1e9-56bfcc87ff-9j7vl" event={"ID":"ffc08e84-a6f3-4d36-8969-c83a498924aa","Type":"ContainerStarted","Data":"c6f016ec366c6decf22fb9b5bfce90f34642f2d3a0475ee5f3c70af7241c5731"} Sep 09 10:44:44.765912 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:44.765843 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-40f1e9-56bfcc87ff-9j7vl" event={"ID":"ffc08e84-a6f3-4d36-8969-c83a498924aa","Type":"ContainerStarted","Data":"896f9600c61ea7e3c7836a049ec2a31e33d4a448174fc637421f70bad8bb2a01"} Sep 09 10:44:44.765955 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:44.765917 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-40f1e9-56bfcc87ff-9j7vl" Sep 09 10:44:44.785151 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:44.785091 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40f1e9-5d4c4cf546-qvvtb" podStartSLOduration=1.785067879 podStartE2EDuration="1.785067879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:44:44.783040249 +0000 UTC m=+1124.957642575" watchObservedRunningTime="2026-09-09 10:44:44.785067879 +0000 UTC m=+1124.959670208" Sep 09 10:44:44.803635 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:44.803575 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-40f1e9-56bfcc87ff-9j7vl" podStartSLOduration=1.8035570669999998 podStartE2EDuration="1.803557067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:44:44.803035237 +0000 UTC m=+1124.977637575" watchObservedRunningTime="2026-09-09 10:44:44.803557067 +0000 UTC m=+1124.978159395" Sep 09 10:44:51.859814 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:51.859777 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e52855bf-5d9f89978-99cbr"] Sep 09 10:44:51.863175 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:51.863157 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e52855bf-5d9f89978-99cbr" Sep 09 10:44:51.866017 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:51.865992 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e52855bf-dockercfg-ff259\"" Sep 09 10:44:51.866191 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:51.866169 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e52855bf\"" Sep 09 10:44:51.880995 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:51.880965 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e52855bf-5d9f89978-99cbr"] Sep 09 10:44:51.992178 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:51.992142 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e52855bf-764dbdcc55-blrb7"] Sep 09 10:44:51.995405 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:51.995389 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e52855bf-764dbdcc55-blrb7" Sep 09 10:44:52.000725 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:52.000691 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn8f4\" (UniqueName: \"kubernetes.io/projected/ea38e680-6641-4bc7-bf7d-69706d86e950-kube-api-access-jn8f4\") pod \"payload-pre-processing-e2e-ait-e52855bf-5d9f89978-99cbr\" (UID: \"ea38e680-6641-4bc7-bf7d-69706d86e950\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e52855bf-5d9f89978-99cbr" Sep 09 10:44:52.000858 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:52.000735 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea38e680-6641-4bc7-bf7d-69706d86e950-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e52855bf-5d9f89978-99cbr\" (UID: \"ea38e680-6641-4bc7-bf7d-69706d86e950\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e52855bf-5d9f89978-99cbr" Sep 09 10:44:52.038339 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:52.038302 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e52855bf-764dbdcc55-blrb7"] Sep 09 10:44:52.102236 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:52.102135 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn8f4\" (UniqueName: \"kubernetes.io/projected/ea38e680-6641-4bc7-bf7d-69706d86e950-kube-api-access-jn8f4\") pod \"payload-pre-processing-e2e-ait-e52855bf-5d9f89978-99cbr\" (UID: \"ea38e680-6641-4bc7-bf7d-69706d86e950\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e52855bf-5d9f89978-99cbr" Sep 09 10:44:52.102413 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:52.102279 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea38e680-6641-4bc7-bf7d-69706d86e950-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e52855bf-5d9f89978-99cbr\" (UID: \"ea38e680-6641-4bc7-bf7d-69706d86e950\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e52855bf-5d9f89978-99cbr" Sep 09 10:44:52.102413 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:52.102359 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e581076-57d8-4a7a-abdc-c9998175647d-plugins-config-volume\") pod \"payload-processing-e2e-ait-e52855bf-764dbdcc55-blrb7\" (UID: \"7e581076-57d8-4a7a-abdc-c9998175647d\") " pod="openshift-ingress/payload-processing-e2e-ait-e52855bf-764dbdcc55-blrb7" Sep 09 10:44:52.102413 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:52.102386 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhwph\" (UniqueName: \"kubernetes.io/projected/7e581076-57d8-4a7a-abdc-c9998175647d-kube-api-access-zhwph\") pod \"payload-processing-e2e-ait-e52855bf-764dbdcc55-blrb7\" (UID: \"7e581076-57d8-4a7a-abdc-c9998175647d\") " pod="openshift-ingress/payload-processing-e2e-ait-e52855bf-764dbdcc55-blrb7" Sep 09 10:44:52.103048 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:52.103026 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea38e680-6641-4bc7-bf7d-69706d86e950-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e52855bf-5d9f89978-99cbr\" (UID: \"ea38e680-6641-4bc7-bf7d-69706d86e950\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e52855bf-5d9f89978-99cbr" Sep 09 10:44:52.121713 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:52.121647 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn8f4\" (UniqueName: \"kubernetes.io/projected/ea38e680-6641-4bc7-bf7d-69706d86e950-kube-api-access-jn8f4\") pod \"payload-pre-processing-e2e-ait-e52855bf-5d9f89978-99cbr\" (UID: \"ea38e680-6641-4bc7-bf7d-69706d86e950\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e52855bf-5d9f89978-99cbr" Sep 09 10:44:52.174897 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:52.174851 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e52855bf-5d9f89978-99cbr" Sep 09 10:44:52.203426 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:52.203395 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e581076-57d8-4a7a-abdc-c9998175647d-plugins-config-volume\") pod \"payload-processing-e2e-ait-e52855bf-764dbdcc55-blrb7\" (UID: \"7e581076-57d8-4a7a-abdc-c9998175647d\") " pod="openshift-ingress/payload-processing-e2e-ait-e52855bf-764dbdcc55-blrb7" Sep 09 10:44:52.203589 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:52.203431 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhwph\" (UniqueName: \"kubernetes.io/projected/7e581076-57d8-4a7a-abdc-c9998175647d-kube-api-access-zhwph\") pod \"payload-processing-e2e-ait-e52855bf-764dbdcc55-blrb7\" (UID: \"7e581076-57d8-4a7a-abdc-c9998175647d\") " pod="openshift-ingress/payload-processing-e2e-ait-e52855bf-764dbdcc55-blrb7" Sep 09 10:44:52.204097 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:52.204076 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e581076-57d8-4a7a-abdc-c9998175647d-plugins-config-volume\") pod \"payload-processing-e2e-ait-e52855bf-764dbdcc55-blrb7\" (UID: \"7e581076-57d8-4a7a-abdc-c9998175647d\") " pod="openshift-ingress/payload-processing-e2e-ait-e52855bf-764dbdcc55-blrb7" Sep 09 10:44:52.213275 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:52.213245 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhwph\" (UniqueName: \"kubernetes.io/projected/7e581076-57d8-4a7a-abdc-c9998175647d-kube-api-access-zhwph\") pod \"payload-processing-e2e-ait-e52855bf-764dbdcc55-blrb7\" (UID: \"7e581076-57d8-4a7a-abdc-c9998175647d\") " pod="openshift-ingress/payload-processing-e2e-ait-e52855bf-764dbdcc55-blrb7" Sep 09 10:44:52.304913 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:52.304879 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e52855bf-764dbdcc55-blrb7" Sep 09 10:44:52.453466 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:44:52.453435 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea38e680_6641_4bc7_bf7d_69706d86e950.slice/crio-e1e5f2199c70407e432e5cb0d62519daaf9d1ca361d02b8773d96047c3e35462 WatchSource:0}: Error finding container e1e5f2199c70407e432e5cb0d62519daaf9d1ca361d02b8773d96047c3e35462: Status 404 returned error can't find the container with id e1e5f2199c70407e432e5cb0d62519daaf9d1ca361d02b8773d96047c3e35462 Sep 09 10:44:52.455323 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:52.455296 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e52855bf-5d9f89978-99cbr"] Sep 09 10:44:52.586999 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:52.586973 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e52855bf-764dbdcc55-blrb7"] Sep 09 10:44:52.589781 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:44:52.589742 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e581076_57d8_4a7a_abdc_c9998175647d.slice/crio-720f9b5231e45f6f67e08f6a10cbd4db5add7aa4409aff0bf1ad0c21a0573dd0 WatchSource:0}: Error finding container 720f9b5231e45f6f67e08f6a10cbd4db5add7aa4409aff0bf1ad0c21a0573dd0: Status 404 returned error can't find the container with id 720f9b5231e45f6f67e08f6a10cbd4db5add7aa4409aff0bf1ad0c21a0573dd0 Sep 09 10:44:52.805727 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:52.805596 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e52855bf-5d9f89978-99cbr" event={"ID":"ea38e680-6641-4bc7-bf7d-69706d86e950","Type":"ContainerStarted","Data":"a07484e28b4ea21ae78c191b083f3d565fc1699dc43e630342fe4c4c64ba6c37"} Sep 09 10:44:52.805727 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:52.805661 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e52855bf-5d9f89978-99cbr" event={"ID":"ea38e680-6641-4bc7-bf7d-69706d86e950","Type":"ContainerStarted","Data":"e1e5f2199c70407e432e5cb0d62519daaf9d1ca361d02b8773d96047c3e35462"} Sep 09 10:44:52.806010 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:52.805731 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e52855bf-5d9f89978-99cbr" Sep 09 10:44:52.807246 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:52.807222 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e52855bf-764dbdcc55-blrb7" event={"ID":"7e581076-57d8-4a7a-abdc-c9998175647d","Type":"ContainerStarted","Data":"11da920d5c01fdf586ef93d14b88c9733174d5b2a6202f1dcfc2841e2184bbd5"} Sep 09 10:44:52.807373 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:52.807262 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e52855bf-764dbdcc55-blrb7" event={"ID":"7e581076-57d8-4a7a-abdc-c9998175647d","Type":"ContainerStarted","Data":"720f9b5231e45f6f67e08f6a10cbd4db5add7aa4409aff0bf1ad0c21a0573dd0"} Sep 09 10:44:52.807373 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:52.807319 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e52855bf-764dbdcc55-blrb7" Sep 09 10:44:52.828202 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:52.828149 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e52855bf-5d9f89978-99cbr" podStartSLOduration=1.8281341100000001 podStartE2EDuration="1.82813411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:44:52.826945528 +0000 UTC m=+1133.001547856" watchObservedRunningTime="2026-09-09 10:44:52.82813411 +0000 UTC m=+1133.002736434" Sep 09 10:44:53.894750 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:53.894693 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e52855bf-764dbdcc55-blrb7" podStartSLOduration=2.894673763 podStartE2EDuration="2.894673763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:44:52.846783545 +0000 UTC m=+1133.021385871" watchObservedRunningTime="2026-09-09 10:44:53.894673763 +0000 UTC m=+1134.069276088" Sep 09 10:44:53.896028 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:53.896008 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4bbfe3-c47867b4b-z6qd8"] Sep 09 10:44:53.899523 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:53.899481 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-4bbfe3-c47867b4b-z6qd8" Sep 09 10:44:53.902061 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:53.902037 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-4bbfe3-dockercfg-2rqjw\"" Sep 09 10:44:53.902178 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:53.902056 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-4bbfe3\"" Sep 09 10:44:53.909967 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:53.909941 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4bbfe3-c47867b4b-z6qd8"] Sep 09 10:44:54.020319 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:54.020283 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9vm9\" (UniqueName: \"kubernetes.io/projected/659bf760-b7e1-4a04-96b5-d33c3acacb59-kube-api-access-l9vm9\") pod \"payload-pre-processing-e2e-autores-4bbfe3-c47867b4b-z6qd8\" (UID: \"659bf760-b7e1-4a04-96b5-d33c3acacb59\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4bbfe3-c47867b4b-z6qd8" Sep 09 10:44:54.020465 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:54.020339 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/659bf760-b7e1-4a04-96b5-d33c3acacb59-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-4bbfe3-c47867b4b-z6qd8\" (UID: \"659bf760-b7e1-4a04-96b5-d33c3acacb59\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4bbfe3-c47867b4b-z6qd8" Sep 09 10:44:54.120937 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:54.120895 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/659bf760-b7e1-4a04-96b5-d33c3acacb59-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-4bbfe3-c47867b4b-z6qd8\" (UID: \"659bf760-b7e1-4a04-96b5-d33c3acacb59\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4bbfe3-c47867b4b-z6qd8" Sep 09 10:44:54.121110 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:54.121012 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9vm9\" (UniqueName: \"kubernetes.io/projected/659bf760-b7e1-4a04-96b5-d33c3acacb59-kube-api-access-l9vm9\") pod \"payload-pre-processing-e2e-autores-4bbfe3-c47867b4b-z6qd8\" (UID: \"659bf760-b7e1-4a04-96b5-d33c3acacb59\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4bbfe3-c47867b4b-z6qd8" Sep 09 10:44:54.121756 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:54.121737 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/659bf760-b7e1-4a04-96b5-d33c3acacb59-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-4bbfe3-c47867b4b-z6qd8\" (UID: \"659bf760-b7e1-4a04-96b5-d33c3acacb59\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4bbfe3-c47867b4b-z6qd8" Sep 09 10:44:54.130371 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:54.130349 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9vm9\" (UniqueName: \"kubernetes.io/projected/659bf760-b7e1-4a04-96b5-d33c3acacb59-kube-api-access-l9vm9\") pod \"payload-pre-processing-e2e-autores-4bbfe3-c47867b4b-z6qd8\" (UID: \"659bf760-b7e1-4a04-96b5-d33c3acacb59\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4bbfe3-c47867b4b-z6qd8" Sep 09 10:44:54.211598 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:54.211482 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-4bbfe3-c47867b4b-z6qd8" Sep 09 10:44:54.491128 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:54.491021 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4bbfe3-c47867b4b-z6qd8"] Sep 09 10:44:54.494953 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:44:54.494927 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod659bf760_b7e1_4a04_96b5_d33c3acacb59.slice/crio-5135f5fe5ad8baf8a50371a656366958dbc8b6da522ba66de0fd46e61ae76dca WatchSource:0}: Error finding container 5135f5fe5ad8baf8a50371a656366958dbc8b6da522ba66de0fd46e61ae76dca: Status 404 returned error can't find the container with id 5135f5fe5ad8baf8a50371a656366958dbc8b6da522ba66de0fd46e61ae76dca Sep 09 10:44:54.819907 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:54.819814 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-4bbfe3-c47867b4b-z6qd8" event={"ID":"659bf760-b7e1-4a04-96b5-d33c3acacb59","Type":"ContainerStarted","Data":"b5b57c67d6c1964fb21a1f9ee21b0c7054e63b2a35e06968f9bb43811f59cbc5"} Sep 09 10:44:54.819907 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:54.819871 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-4bbfe3-c47867b4b-z6qd8" event={"ID":"659bf760-b7e1-4a04-96b5-d33c3acacb59","Type":"ContainerStarted","Data":"5135f5fe5ad8baf8a50371a656366958dbc8b6da522ba66de0fd46e61ae76dca"} Sep 09 10:44:54.819907 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:54.819895 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-4bbfe3-c47867b4b-z6qd8" Sep 09 10:44:55.773295 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:55.773261 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40f1e9-5d4c4cf546-qvvtb" Sep 09 10:44:55.773738 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:55.773702 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-40f1e9-56bfcc87ff-9j7vl" Sep 09 10:44:55.801963 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:44:55.801907 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-4bbfe3-c47867b4b-z6qd8" podStartSLOduration=2.801891634 podStartE2EDuration="2.801891634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:44:54.848845298 +0000 UTC m=+1135.023447622" watchObservedRunningTime="2026-09-09 10:44:55.801891634 +0000 UTC m=+1135.976493958" Sep 09 10:45:00.172016 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:00.171981 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815845-276hk"] Sep 09 10:45:00.175132 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:00.175108 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e52855bf-29815845-f6hvs"] Sep 09 10:45:00.175253 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:00.175229 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815845-276hk" Sep 09 10:45:00.178471 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:00.178448 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kq7qv\"" Sep 09 10:45:00.178717 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:00.178700 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e52855bf-29815845-f6hvs" Sep 09 10:45:00.189782 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:00.189755 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kq7qv\"" Sep 09 10:45:00.192918 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:00.192899 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e52855bf-29815845-f6hvs" Sep 09 10:45:00.196531 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:00.196487 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815845-276hk" Sep 09 10:45:00.200653 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:00.200628 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e52855bf-29815845-f6hvs"] Sep 09 10:45:00.218019 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:00.217985 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815845-276hk"] Sep 09 10:45:00.522573 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:00.522543 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815845-276hk"] Sep 09 10:45:00.524275 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:45:00.524242 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a594f7_1a3f_4e60_b0ea_69ceb6221560.slice/crio-9d348faa1767c6006038815a04c92804781851e8fbd42341866d84b3bc16395f WatchSource:0}: Error finding container 9d348faa1767c6006038815a04c92804781851e8fbd42341866d84b3bc16395f: Status 404 returned error can't find the container with id 9d348faa1767c6006038815a04c92804781851e8fbd42341866d84b3bc16395f Sep 09 10:45:00.524406 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:00.524324 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e52855bf-29815845-f6hvs"] Sep 09 10:45:00.524791 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:45:00.524770 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd01a9ecb_d63d_4fff_942f_4f0c355a79f3.slice/crio-b2f7a4df6bcb25c160bfc03940a01a133db7cc303842afd659dc5023a4f29dd5 WatchSource:0}: Error finding container b2f7a4df6bcb25c160bfc03940a01a133db7cc303842afd659dc5023a4f29dd5: Status 404 returned error can't find the container with id b2f7a4df6bcb25c160bfc03940a01a133db7cc303842afd659dc5023a4f29dd5 Sep 09 10:45:00.854122 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:00.854084 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e52855bf-29815845-f6hvs" event={"ID":"d01a9ecb-d63d-4fff-942f-4f0c355a79f3","Type":"ContainerStarted","Data":"b2f7a4df6bcb25c160bfc03940a01a133db7cc303842afd659dc5023a4f29dd5"} Sep 09 10:45:00.855433 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:00.855408 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815845-276hk" event={"ID":"19a594f7-1a3f-4e60-b0ea-69ceb6221560","Type":"ContainerStarted","Data":"9d348faa1767c6006038815a04c92804781851e8fbd42341866d84b3bc16395f"} Sep 09 10:45:03.814258 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:03.814171 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e52855bf-764dbdcc55-blrb7" Sep 09 10:45:03.814258 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:03.814257 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e52855bf-5d9f89978-99cbr" Sep 09 10:45:03.874409 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:03.874373 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e52855bf-29815845-f6hvs" event={"ID":"d01a9ecb-d63d-4fff-942f-4f0c355a79f3","Type":"ContainerStarted","Data":"24e7abcb3783c4d877f8e627b9898ee6bd6fbbb0dc351f58bf3f7f2ed24dd0d8"} Sep 09 10:45:03.876217 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:03.876182 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815845-276hk" event={"ID":"19a594f7-1a3f-4e60-b0ea-69ceb6221560","Type":"ContainerStarted","Data":"c2c5b35829c342575d13db73157649c4368519b6e4a3c4b283c6cbb433d12a08"} Sep 09 10:45:03.895437 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:03.895372 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e52855bf-29815845-f6hvs" podStartSLOduration=1.18781056 podStartE2EDuration="3.895354221s" totalImagesPullingTime="2.707543661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:45:03.893363178 +0000 UTC m=+1144.067965505" watchObservedRunningTime="2026-09-09 10:45:03.895354221 +0000 UTC m=+1144.069956546" Sep 09 10:45:04.882963 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:04.882922 2559 generic.go:356] "Generic (PLEG): container finished" podID="d01a9ecb-d63d-4fff-942f-4f0c355a79f3" containerID="24e7abcb3783c4d877f8e627b9898ee6bd6fbbb0dc351f58bf3f7f2ed24dd0d8" exitCode=0 Sep 09 10:45:04.883382 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:04.883004 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e52855bf-29815845-f6hvs" event={"ID":"d01a9ecb-d63d-4fff-942f-4f0c355a79f3","Type":"ContainerDied","Data":"24e7abcb3783c4d877f8e627b9898ee6bd6fbbb0dc351f58bf3f7f2ed24dd0d8"} Sep 09 10:45:04.884569 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:04.884547 2559 generic.go:356] "Generic (PLEG): container finished" podID="19a594f7-1a3f-4e60-b0ea-69ceb6221560" containerID="c2c5b35829c342575d13db73157649c4368519b6e4a3c4b283c6cbb433d12a08" exitCode=0 Sep 09 10:45:04.884654 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:04.884631 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815845-276hk" event={"ID":"19a594f7-1a3f-4e60-b0ea-69ceb6221560","Type":"ContainerDied","Data":"c2c5b35829c342575d13db73157649c4368519b6e4a3c4b283c6cbb433d12a08"} Sep 09 10:45:04.909111 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:04.909060 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815845-276hk" podStartSLOduration=2.196536356 podStartE2EDuration="4.909040405s" totalImagesPullingTime="2.712504049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:45:03.910828339 +0000 UTC m=+1144.085430666" watchObservedRunningTime="2026-09-09 10:45:04.909040405 +0000 UTC m=+1145.083642731" Sep 09 10:45:05.829061 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:05.829032 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-4bbfe3-c47867b4b-z6qd8" Sep 09 10:45:06.052895 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:06.052867 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815845-276hk" Sep 09 10:45:06.061847 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:06.061823 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e52855bf-29815845-f6hvs" Sep 09 10:45:06.896932 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:06.896897 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815845-276hk" Sep 09 10:45:06.897121 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:06.896931 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815845-276hk" event={"ID":"19a594f7-1a3f-4e60-b0ea-69ceb6221560","Type":"ContainerDied","Data":"9d348faa1767c6006038815a04c92804781851e8fbd42341866d84b3bc16395f"} Sep 09 10:45:06.897121 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:06.896980 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d348faa1767c6006038815a04c92804781851e8fbd42341866d84b3bc16395f" Sep 09 10:45:06.898364 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:06.898341 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e52855bf-29815845-f6hvs" event={"ID":"d01a9ecb-d63d-4fff-942f-4f0c355a79f3","Type":"ContainerDied","Data":"b2f7a4df6bcb25c160bfc03940a01a133db7cc303842afd659dc5023a4f29dd5"} Sep 09 10:45:06.898470 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:06.898383 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2f7a4df6bcb25c160bfc03940a01a133db7cc303842afd659dc5023a4f29dd5" Sep 09 10:45:06.898470 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:06.898384 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e52855bf-29815845-f6hvs" Sep 09 10:45:12.133823 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.133791 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-da3c64-78894768f-xfdwn"] Sep 09 10:45:12.134210 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.134153 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d01a9ecb-d63d-4fff-942f-4f0c355a79f3" containerName="cleanup" Sep 09 10:45:12.134210 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.134164 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d01a9ecb-d63d-4fff-942f-4f0c355a79f3" containerName="cleanup" Sep 09 10:45:12.134210 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.134189 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19a594f7-1a3f-4e60-b0ea-69ceb6221560" containerName="cleanup" Sep 09 10:45:12.134210 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.134195 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="19a594f7-1a3f-4e60-b0ea-69ceb6221560" containerName="cleanup" Sep 09 10:45:12.134333 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.134249 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="19a594f7-1a3f-4e60-b0ea-69ceb6221560" containerName="cleanup" Sep 09 10:45:12.134333 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.134259 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d01a9ecb-d63d-4fff-942f-4f0c355a79f3" containerName="cleanup" Sep 09 10:45:12.137549 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.137522 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-da3c64-644699bc86-86wjk"] Sep 09 10:45:12.137685 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.137579 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-da3c64-78894768f-xfdwn" Sep 09 10:45:12.140769 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.140744 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-da3c64\"" Sep 09 10:45:12.140769 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.140770 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da3c64-644699bc86-86wjk" Sep 09 10:45:12.141890 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.141868 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-da3c64-dockercfg-fc79k\"" Sep 09 10:45:12.147973 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.147946 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-da3c64-78894768f-xfdwn"] Sep 09 10:45:12.151257 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.151232 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-da3c64-644699bc86-86wjk"] Sep 09 10:45:12.285622 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.285586 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bkbn\" (UniqueName: \"kubernetes.io/projected/d7218d2e-5072-4134-a5d4-e5865febfe64-kube-api-access-6bkbn\") pod \"payload-pre-processing-e2e-inf-b-da3c64-644699bc86-86wjk\" (UID: \"d7218d2e-5072-4134-a5d4-e5865febfe64\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da3c64-644699bc86-86wjk" Sep 09 10:45:12.285831 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.285639 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7218d2e-5072-4134-a5d4-e5865febfe64-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-da3c64-644699bc86-86wjk\" (UID: \"d7218d2e-5072-4134-a5d4-e5865febfe64\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da3c64-644699bc86-86wjk" Sep 09 10:45:12.285831 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.285663 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bbb87f1-9446-4f65-a9aa-85704da050e9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-da3c64-78894768f-xfdwn\" (UID: \"9bbb87f1-9446-4f65-a9aa-85704da050e9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-da3c64-78894768f-xfdwn" Sep 09 10:45:12.285831 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.285741 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6nds\" (UniqueName: \"kubernetes.io/projected/9bbb87f1-9446-4f65-a9aa-85704da050e9-kube-api-access-s6nds\") pod \"payload-processing-e2e-inf-b-da3c64-78894768f-xfdwn\" (UID: \"9bbb87f1-9446-4f65-a9aa-85704da050e9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-da3c64-78894768f-xfdwn" Sep 09 10:45:12.387275 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.387182 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6nds\" (UniqueName: \"kubernetes.io/projected/9bbb87f1-9446-4f65-a9aa-85704da050e9-kube-api-access-s6nds\") pod \"payload-processing-e2e-inf-b-da3c64-78894768f-xfdwn\" (UID: \"9bbb87f1-9446-4f65-a9aa-85704da050e9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-da3c64-78894768f-xfdwn" Sep 09 10:45:12.387275 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.387248 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bkbn\" (UniqueName: \"kubernetes.io/projected/d7218d2e-5072-4134-a5d4-e5865febfe64-kube-api-access-6bkbn\") pod \"payload-pre-processing-e2e-inf-b-da3c64-644699bc86-86wjk\" (UID: \"d7218d2e-5072-4134-a5d4-e5865febfe64\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da3c64-644699bc86-86wjk" Sep 09 10:45:12.387275 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.387276 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7218d2e-5072-4134-a5d4-e5865febfe64-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-da3c64-644699bc86-86wjk\" (UID: \"d7218d2e-5072-4134-a5d4-e5865febfe64\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da3c64-644699bc86-86wjk" Sep 09 10:45:12.387513 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.387298 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bbb87f1-9446-4f65-a9aa-85704da050e9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-da3c64-78894768f-xfdwn\" (UID: \"9bbb87f1-9446-4f65-a9aa-85704da050e9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-da3c64-78894768f-xfdwn" Sep 09 10:45:12.388040 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.388017 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bbb87f1-9446-4f65-a9aa-85704da050e9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-da3c64-78894768f-xfdwn\" (UID: \"9bbb87f1-9446-4f65-a9aa-85704da050e9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-da3c64-78894768f-xfdwn" Sep 09 10:45:12.388081 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.388028 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7218d2e-5072-4134-a5d4-e5865febfe64-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-da3c64-644699bc86-86wjk\" (UID: \"d7218d2e-5072-4134-a5d4-e5865febfe64\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da3c64-644699bc86-86wjk" Sep 09 10:45:12.396822 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.396790 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6nds\" (UniqueName: \"kubernetes.io/projected/9bbb87f1-9446-4f65-a9aa-85704da050e9-kube-api-access-s6nds\") pod \"payload-processing-e2e-inf-b-da3c64-78894768f-xfdwn\" (UID: \"9bbb87f1-9446-4f65-a9aa-85704da050e9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-da3c64-78894768f-xfdwn" Sep 09 10:45:12.396928 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.396853 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bkbn\" (UniqueName: \"kubernetes.io/projected/d7218d2e-5072-4134-a5d4-e5865febfe64-kube-api-access-6bkbn\") pod \"payload-pre-processing-e2e-inf-b-da3c64-644699bc86-86wjk\" (UID: \"d7218d2e-5072-4134-a5d4-e5865febfe64\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da3c64-644699bc86-86wjk" Sep 09 10:45:12.450624 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.450587 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-da3c64-78894768f-xfdwn" Sep 09 10:45:12.459719 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.459688 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da3c64-644699bc86-86wjk" Sep 09 10:45:12.728061 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.727975 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-da3c64-78894768f-xfdwn"] Sep 09 10:45:12.731623 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:45:12.731575 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bbb87f1_9446_4f65_a9aa_85704da050e9.slice/crio-ecdfadbd23fa40787b0f5668251c738271850c5b21349851e0dda1fce4cfccb7 WatchSource:0}: Error finding container ecdfadbd23fa40787b0f5668251c738271850c5b21349851e0dda1fce4cfccb7: Status 404 returned error can't find the container with id ecdfadbd23fa40787b0f5668251c738271850c5b21349851e0dda1fce4cfccb7 Sep 09 10:45:12.746566 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.746538 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-da3c64-644699bc86-86wjk"] Sep 09 10:45:12.749602 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:45:12.749572 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7218d2e_5072_4134_a5d4_e5865febfe64.slice/crio-2f3f9a12bcd35cb9a19154afb9c67564ebf6555498bd16feeda8ca5bb66128f3 WatchSource:0}: Error finding container 2f3f9a12bcd35cb9a19154afb9c67564ebf6555498bd16feeda8ca5bb66128f3: Status 404 returned error can't find the container with id 2f3f9a12bcd35cb9a19154afb9c67564ebf6555498bd16feeda8ca5bb66128f3 Sep 09 10:45:12.931092 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.931048 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-da3c64-78894768f-xfdwn" event={"ID":"9bbb87f1-9446-4f65-a9aa-85704da050e9","Type":"ContainerStarted","Data":"c5f8c568a3f4dc94b25896fad3421f7798d3a5908f578e65cc653ee949b26f1f"} Sep 09 10:45:12.931295 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.931148 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-da3c64-78894768f-xfdwn" event={"ID":"9bbb87f1-9446-4f65-a9aa-85704da050e9","Type":"ContainerStarted","Data":"ecdfadbd23fa40787b0f5668251c738271850c5b21349851e0dda1fce4cfccb7"} Sep 09 10:45:12.931295 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.931277 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-da3c64-78894768f-xfdwn" Sep 09 10:45:12.932650 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.932625 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da3c64-644699bc86-86wjk" event={"ID":"d7218d2e-5072-4134-a5d4-e5865febfe64","Type":"ContainerStarted","Data":"52d06511ba69f1cec744aa5ba9fe069f7e6ebedd5c27077b6a78195b546364e7"} Sep 09 10:45:12.932763 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.932663 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da3c64-644699bc86-86wjk" event={"ID":"d7218d2e-5072-4134-a5d4-e5865febfe64","Type":"ContainerStarted","Data":"2f3f9a12bcd35cb9a19154afb9c67564ebf6555498bd16feeda8ca5bb66128f3"} Sep 09 10:45:12.932763 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.932715 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da3c64-644699bc86-86wjk" Sep 09 10:45:12.950410 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.950356 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-da3c64-78894768f-xfdwn" podStartSLOduration=0.950337677 podStartE2EDuration="950.337677ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:45:12.949893346 +0000 UTC m=+1153.124495672" watchObservedRunningTime="2026-09-09 10:45:12.950337677 +0000 UTC m=+1153.124940004" Sep 09 10:45:12.967318 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:12.967260 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da3c64-644699bc86-86wjk" podStartSLOduration=0.967241502 podStartE2EDuration="967.241502ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:45:12.966070116 +0000 UTC m=+1153.140672443" watchObservedRunningTime="2026-09-09 10:45:12.967241502 +0000 UTC m=+1153.141843827" Sep 09 10:45:15.182495 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:15.182456 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52855bf-bps2d"] Sep 09 10:45:15.187293 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:15.187274 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52855bf-bps2d" Sep 09 10:45:15.190609 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:15.190588 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kq7qv\"" Sep 09 10:45:15.191366 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:15.191340 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 10:45:15.203590 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:15.203551 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52855bf-bps2d"] Sep 09 10:45:15.313693 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:15.313654 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03019926-d7b4-4e46-80ff-17db3161d053-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e52855bf-bps2d\" (UID: \"03019926-d7b4-4e46-80ff-17db3161d053\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52855bf-bps2d" Sep 09 10:45:15.415038 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:15.414999 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03019926-d7b4-4e46-80ff-17db3161d053-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e52855bf-bps2d\" (UID: \"03019926-d7b4-4e46-80ff-17db3161d053\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52855bf-bps2d" Sep 09 10:45:15.415716 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:15.415692 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03019926-d7b4-4e46-80ff-17db3161d053-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e52855bf-bps2d\" (UID: \"03019926-d7b4-4e46-80ff-17db3161d053\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52855bf-bps2d" Sep 09 10:45:15.499622 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:15.499541 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52855bf-bps2d" Sep 09 10:45:15.777154 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:15.777067 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52855bf-bps2d"] Sep 09 10:45:15.780810 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:45:15.780784 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03019926_d7b4_4e46_80ff_17db3161d053.slice/crio-b2cfcfd218d225e4fc88c5d5d9dcf89ece7ab3b08cc4207a5235e70ed449b52e WatchSource:0}: Error finding container b2cfcfd218d225e4fc88c5d5d9dcf89ece7ab3b08cc4207a5235e70ed449b52e: Status 404 returned error can't find the container with id b2cfcfd218d225e4fc88c5d5d9dcf89ece7ab3b08cc4207a5235e70ed449b52e Sep 09 10:45:15.950775 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:15.950744 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52855bf-bps2d" event={"ID":"03019926-d7b4-4e46-80ff-17db3161d053","Type":"ContainerStarted","Data":"b2cfcfd218d225e4fc88c5d5d9dcf89ece7ab3b08cc4207a5235e70ed449b52e"} Sep 09 10:45:16.957997 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:16.957906 2559 generic.go:356] "Generic (PLEG): container finished" podID="03019926-d7b4-4e46-80ff-17db3161d053" containerID="7598fc17d5d476ea2585ab4f6f923adbba2a9393a124964a300b79c19d9df6a6" exitCode=0 Sep 09 10:45:16.957997 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:16.957942 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52855bf-bps2d" event={"ID":"03019926-d7b4-4e46-80ff-17db3161d053","Type":"ContainerDied","Data":"7598fc17d5d476ea2585ab4f6f923adbba2a9393a124964a300b79c19d9df6a6"} Sep 09 10:45:18.098138 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:18.098111 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52855bf-bps2d" Sep 09 10:45:18.242458 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:18.242318 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03019926-d7b4-4e46-80ff-17db3161d053-maas-api-service-ca\") pod \"03019926-d7b4-4e46-80ff-17db3161d053\" (UID: \"03019926-d7b4-4e46-80ff-17db3161d053\") " Sep 09 10:45:18.242791 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:18.242765 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03019926-d7b4-4e46-80ff-17db3161d053-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "03019926-d7b4-4e46-80ff-17db3161d053" (UID: "03019926-d7b4-4e46-80ff-17db3161d053"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:45:18.343682 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:18.343646 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03019926-d7b4-4e46-80ff-17db3161d053-maas-api-service-ca\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:45:18.969640 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:18.969596 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52855bf-bps2d" event={"ID":"03019926-d7b4-4e46-80ff-17db3161d053","Type":"ContainerDied","Data":"b2cfcfd218d225e4fc88c5d5d9dcf89ece7ab3b08cc4207a5235e70ed449b52e"} Sep 09 10:45:18.969813 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:18.969649 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52855bf-bps2d" Sep 09 10:45:18.969813 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:18.969660 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2cfcfd218d225e4fc88c5d5d9dcf89ece7ab3b08cc4207a5235e70ed449b52e" Sep 09 10:45:21.835092 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:21.835052 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e52855bf-29815845-f6hvs"] Sep 09 10:45:21.837269 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:21.837247 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e52855bf-29815845-f6hvs"] Sep 09 10:45:21.867836 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:21.867803 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e52855bf-764dbdcc55-blrb7"] Sep 09 10:45:21.868135 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:21.868079 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e52855bf-764dbdcc55-blrb7" podUID="7e581076-57d8-4a7a-abdc-c9998175647d" containerName="payload-processing" containerID="cri-o://11da920d5c01fdf586ef93d14b88c9733174d5b2a6202f1dcfc2841e2184bbd5" gracePeriod=30 Sep 09 10:45:21.880343 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:21.880314 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e52855bf-5d9f89978-99cbr"] Sep 09 10:45:21.880614 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:21.880568 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e52855bf-5d9f89978-99cbr" podUID="ea38e680-6641-4bc7-bf7d-69706d86e950" containerName="payload-pre-processing" containerID="cri-o://a07484e28b4ea21ae78c191b083f3d565fc1699dc43e630342fe4c4c64ba6c37" gracePeriod=30 Sep 09 10:45:21.986828 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:21.986718 2559 generic.go:356] "Generic (PLEG): container finished" podID="7e581076-57d8-4a7a-abdc-c9998175647d" containerID="11da920d5c01fdf586ef93d14b88c9733174d5b2a6202f1dcfc2841e2184bbd5" exitCode=0 Sep 09 10:45:21.986905 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:21.986842 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e52855bf-764dbdcc55-blrb7" event={"ID":"7e581076-57d8-4a7a-abdc-c9998175647d","Type":"ContainerDied","Data":"11da920d5c01fdf586ef93d14b88c9733174d5b2a6202f1dcfc2841e2184bbd5"} Sep 09 10:45:22.073737 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:22.073694 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52855bf-bps2d"] Sep 09 10:45:22.077398 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:22.077347 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52855bf-bps2d"] Sep 09 10:45:22.127296 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:22.127273 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e52855bf-764dbdcc55-blrb7" Sep 09 10:45:22.166161 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:22.165839 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e52855bf-5d9f89978-99cbr" Sep 09 10:45:22.282764 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:22.282731 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jn8f4\" (UniqueName: \"kubernetes.io/projected/ea38e680-6641-4bc7-bf7d-69706d86e950-kube-api-access-jn8f4\") pod \"ea38e680-6641-4bc7-bf7d-69706d86e950\" (UID: \"ea38e680-6641-4bc7-bf7d-69706d86e950\") " Sep 09 10:45:22.282932 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:22.282797 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e581076-57d8-4a7a-abdc-c9998175647d-plugins-config-volume\") pod \"7e581076-57d8-4a7a-abdc-c9998175647d\" (UID: \"7e581076-57d8-4a7a-abdc-c9998175647d\") " Sep 09 10:45:22.282932 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:22.282848 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea38e680-6641-4bc7-bf7d-69706d86e950-plugins-config-volume\") pod \"ea38e680-6641-4bc7-bf7d-69706d86e950\" (UID: \"ea38e680-6641-4bc7-bf7d-69706d86e950\") " Sep 09 10:45:22.282932 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:22.282868 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhwph\" (UniqueName: \"kubernetes.io/projected/7e581076-57d8-4a7a-abdc-c9998175647d-kube-api-access-zhwph\") pod \"7e581076-57d8-4a7a-abdc-c9998175647d\" (UID: \"7e581076-57d8-4a7a-abdc-c9998175647d\") " Sep 09 10:45:22.283150 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:22.283117 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e581076-57d8-4a7a-abdc-c9998175647d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7e581076-57d8-4a7a-abdc-c9998175647d" (UID: "7e581076-57d8-4a7a-abdc-c9998175647d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:45:22.283150 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:22.283136 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea38e680-6641-4bc7-bf7d-69706d86e950-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ea38e680-6641-4bc7-bf7d-69706d86e950" (UID: "ea38e680-6641-4bc7-bf7d-69706d86e950"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:45:22.285173 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:22.285142 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea38e680-6641-4bc7-bf7d-69706d86e950-kube-api-access-jn8f4" (OuterVolumeSpecName: "kube-api-access-jn8f4") pod "ea38e680-6641-4bc7-bf7d-69706d86e950" (UID: "ea38e680-6641-4bc7-bf7d-69706d86e950"). InnerVolumeSpecName "kube-api-access-jn8f4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:45:22.285278 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:22.285247 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e581076-57d8-4a7a-abdc-c9998175647d-kube-api-access-zhwph" (OuterVolumeSpecName: "kube-api-access-zhwph") pod "7e581076-57d8-4a7a-abdc-c9998175647d" (UID: "7e581076-57d8-4a7a-abdc-c9998175647d"). InnerVolumeSpecName "kube-api-access-zhwph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:45:22.334739 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:22.334707 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03019926-d7b4-4e46-80ff-17db3161d053" path="/var/lib/kubelet/pods/03019926-d7b4-4e46-80ff-17db3161d053/volumes" Sep 09 10:45:22.334994 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:22.334981 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d01a9ecb-d63d-4fff-942f-4f0c355a79f3" path="/var/lib/kubelet/pods/d01a9ecb-d63d-4fff-942f-4f0c355a79f3/volumes" Sep 09 10:45:22.383867 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:22.383844 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn8f4\" (UniqueName: \"kubernetes.io/projected/ea38e680-6641-4bc7-bf7d-69706d86e950-kube-api-access-jn8f4\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:45:22.383867 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:22.383867 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e581076-57d8-4a7a-abdc-c9998175647d-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:45:22.384003 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:22.383877 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea38e680-6641-4bc7-bf7d-69706d86e950-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:45:22.384003 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:22.383886 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhwph\" (UniqueName: \"kubernetes.io/projected/7e581076-57d8-4a7a-abdc-c9998175647d-kube-api-access-zhwph\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:45:22.992867 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:22.992834 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e52855bf-764dbdcc55-blrb7" Sep 09 10:45:22.993302 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:22.992870 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e52855bf-764dbdcc55-blrb7" event={"ID":"7e581076-57d8-4a7a-abdc-c9998175647d","Type":"ContainerDied","Data":"720f9b5231e45f6f67e08f6a10cbd4db5add7aa4409aff0bf1ad0c21a0573dd0"} Sep 09 10:45:22.993302 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:22.992928 2559 scope.go:117] "RemoveContainer" containerID="11da920d5c01fdf586ef93d14b88c9733174d5b2a6202f1dcfc2841e2184bbd5" Sep 09 10:45:22.994396 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:22.994276 2559 generic.go:356] "Generic (PLEG): container finished" podID="ea38e680-6641-4bc7-bf7d-69706d86e950" containerID="a07484e28b4ea21ae78c191b083f3d565fc1699dc43e630342fe4c4c64ba6c37" exitCode=0 Sep 09 10:45:22.994396 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:22.994311 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e52855bf-5d9f89978-99cbr" event={"ID":"ea38e680-6641-4bc7-bf7d-69706d86e950","Type":"ContainerDied","Data":"a07484e28b4ea21ae78c191b083f3d565fc1699dc43e630342fe4c4c64ba6c37"} Sep 09 10:45:22.994396 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:22.994342 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e52855bf-5d9f89978-99cbr" event={"ID":"ea38e680-6641-4bc7-bf7d-69706d86e950","Type":"ContainerDied","Data":"e1e5f2199c70407e432e5cb0d62519daaf9d1ca361d02b8773d96047c3e35462"} Sep 09 10:45:22.994396 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:22.994314 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e52855bf-5d9f89978-99cbr" Sep 09 10:45:23.002430 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:23.002412 2559 scope.go:117] "RemoveContainer" containerID="a07484e28b4ea21ae78c191b083f3d565fc1699dc43e630342fe4c4c64ba6c37" Sep 09 10:45:23.010339 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:23.010322 2559 scope.go:117] "RemoveContainer" containerID="a07484e28b4ea21ae78c191b083f3d565fc1699dc43e630342fe4c4c64ba6c37" Sep 09 10:45:23.010699 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:45:23.010683 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a07484e28b4ea21ae78c191b083f3d565fc1699dc43e630342fe4c4c64ba6c37\": container with ID starting with a07484e28b4ea21ae78c191b083f3d565fc1699dc43e630342fe4c4c64ba6c37 not found: ID does not exist" containerID="a07484e28b4ea21ae78c191b083f3d565fc1699dc43e630342fe4c4c64ba6c37" Sep 09 10:45:23.010772 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:23.010706 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a07484e28b4ea21ae78c191b083f3d565fc1699dc43e630342fe4c4c64ba6c37"} err="failed to get container status \"a07484e28b4ea21ae78c191b083f3d565fc1699dc43e630342fe4c4c64ba6c37\": rpc error: code = NotFound desc = could not find container \"a07484e28b4ea21ae78c191b083f3d565fc1699dc43e630342fe4c4c64ba6c37\": container with ID starting with a07484e28b4ea21ae78c191b083f3d565fc1699dc43e630342fe4c4c64ba6c37 not found: ID does not exist" Sep 09 10:45:23.011124 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:23.011105 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e52855bf-764dbdcc55-blrb7"] Sep 09 10:45:23.014434 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:23.014403 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e52855bf-764dbdcc55-blrb7"] Sep 09 10:45:23.027231 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:23.027207 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e52855bf-5d9f89978-99cbr"] Sep 09 10:45:23.031693 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:23.031673 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e52855bf-5d9f89978-99cbr"] Sep 09 10:45:23.939109 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:23.939074 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-da3c64-78894768f-xfdwn" Sep 09 10:45:23.939295 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:23.939154 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da3c64-644699bc86-86wjk" Sep 09 10:45:24.340488 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:24.340454 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e581076-57d8-4a7a-abdc-c9998175647d" path="/var/lib/kubelet/pods/7e581076-57d8-4a7a-abdc-c9998175647d/volumes" Sep 09 10:45:24.340864 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:24.340821 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea38e680-6641-4bc7-bf7d-69706d86e950" path="/var/lib/kubelet/pods/ea38e680-6641-4bc7-bf7d-69706d86e950/volumes" Sep 09 10:45:26.868978 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:26.868942 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-92ac8218-6bfbd47d8b-8vxvr"] Sep 09 10:45:26.869348 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:26.869334 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea38e680-6641-4bc7-bf7d-69706d86e950" containerName="payload-pre-processing" Sep 09 10:45:26.869388 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:26.869350 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea38e680-6641-4bc7-bf7d-69706d86e950" containerName="payload-pre-processing" Sep 09 10:45:26.869388 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:26.869362 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03019926-d7b4-4e46-80ff-17db3161d053" containerName="revoke-keys" Sep 09 10:45:26.869388 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:26.869368 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="03019926-d7b4-4e46-80ff-17db3161d053" containerName="revoke-keys" Sep 09 10:45:26.869388 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:26.869386 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e581076-57d8-4a7a-abdc-c9998175647d" containerName="payload-processing" Sep 09 10:45:26.869555 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:26.869392 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e581076-57d8-4a7a-abdc-c9998175647d" containerName="payload-processing" Sep 09 10:45:26.869555 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:26.869447 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="03019926-d7b4-4e46-80ff-17db3161d053" containerName="revoke-keys" Sep 09 10:45:26.869555 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:26.869455 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea38e680-6641-4bc7-bf7d-69706d86e950" containerName="payload-pre-processing" Sep 09 10:45:26.869555 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:26.869464 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e581076-57d8-4a7a-abdc-c9998175647d" containerName="payload-processing" Sep 09 10:45:26.873949 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:26.873931 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-92ac8218-6bfbd47d8b-8vxvr" Sep 09 10:45:26.876776 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:26.876746 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-92ac8218-dockercfg-zbss6\"" Sep 09 10:45:26.877035 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:26.877020 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-92ac8218\"" Sep 09 10:45:26.885425 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:26.885403 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-92ac8218-6bfbd47d8b-8vxvr"] Sep 09 10:45:26.896725 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:26.896702 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-92ac8218-857db98746-dfhfs"] Sep 09 10:45:26.900728 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:26.900708 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-92ac8218-857db98746-dfhfs" Sep 09 10:45:26.910605 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:26.910568 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-92ac8218-857db98746-dfhfs"] Sep 09 10:45:26.920600 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:26.920575 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s69s9\" (UniqueName: \"kubernetes.io/projected/56098277-d76a-4a15-945e-aff99da811cd-kube-api-access-s69s9\") pod \"payload-processing-e2e-migrate-92ac8218-6bfbd47d8b-8vxvr\" (UID: \"56098277-d76a-4a15-945e-aff99da811cd\") " pod="openshift-ingress/payload-processing-e2e-migrate-92ac8218-6bfbd47d8b-8vxvr" Sep 09 10:45:26.920759 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:26.920665 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56098277-d76a-4a15-945e-aff99da811cd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-92ac8218-6bfbd47d8b-8vxvr\" (UID: \"56098277-d76a-4a15-945e-aff99da811cd\") " pod="openshift-ingress/payload-processing-e2e-migrate-92ac8218-6bfbd47d8b-8vxvr" Sep 09 10:45:27.021043 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:27.021012 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s69s9\" (UniqueName: \"kubernetes.io/projected/56098277-d76a-4a15-945e-aff99da811cd-kube-api-access-s69s9\") pod \"payload-processing-e2e-migrate-92ac8218-6bfbd47d8b-8vxvr\" (UID: \"56098277-d76a-4a15-945e-aff99da811cd\") " pod="openshift-ingress/payload-processing-e2e-migrate-92ac8218-6bfbd47d8b-8vxvr" Sep 09 10:45:27.021217 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:27.021069 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcsv9\" (UniqueName: \"kubernetes.io/projected/976b7a1c-81d9-41bc-9071-76c01ec35739-kube-api-access-bcsv9\") pod \"payload-pre-processing-e2e-migrate-92ac8218-857db98746-dfhfs\" (UID: \"976b7a1c-81d9-41bc-9071-76c01ec35739\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-92ac8218-857db98746-dfhfs" Sep 09 10:45:27.021217 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:27.021093 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56098277-d76a-4a15-945e-aff99da811cd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-92ac8218-6bfbd47d8b-8vxvr\" (UID: \"56098277-d76a-4a15-945e-aff99da811cd\") " pod="openshift-ingress/payload-processing-e2e-migrate-92ac8218-6bfbd47d8b-8vxvr" Sep 09 10:45:27.021217 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:27.021194 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976b7a1c-81d9-41bc-9071-76c01ec35739-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-92ac8218-857db98746-dfhfs\" (UID: \"976b7a1c-81d9-41bc-9071-76c01ec35739\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-92ac8218-857db98746-dfhfs" Sep 09 10:45:27.021837 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:27.021814 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56098277-d76a-4a15-945e-aff99da811cd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-92ac8218-6bfbd47d8b-8vxvr\" (UID: \"56098277-d76a-4a15-945e-aff99da811cd\") " pod="openshift-ingress/payload-processing-e2e-migrate-92ac8218-6bfbd47d8b-8vxvr" Sep 09 10:45:27.029655 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:27.029638 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s69s9\" (UniqueName: \"kubernetes.io/projected/56098277-d76a-4a15-945e-aff99da811cd-kube-api-access-s69s9\") pod \"payload-processing-e2e-migrate-92ac8218-6bfbd47d8b-8vxvr\" (UID: \"56098277-d76a-4a15-945e-aff99da811cd\") " pod="openshift-ingress/payload-processing-e2e-migrate-92ac8218-6bfbd47d8b-8vxvr" Sep 09 10:45:27.122123 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:27.122029 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976b7a1c-81d9-41bc-9071-76c01ec35739-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-92ac8218-857db98746-dfhfs\" (UID: \"976b7a1c-81d9-41bc-9071-76c01ec35739\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-92ac8218-857db98746-dfhfs" Sep 09 10:45:27.122266 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:27.122138 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcsv9\" (UniqueName: \"kubernetes.io/projected/976b7a1c-81d9-41bc-9071-76c01ec35739-kube-api-access-bcsv9\") pod \"payload-pre-processing-e2e-migrate-92ac8218-857db98746-dfhfs\" (UID: \"976b7a1c-81d9-41bc-9071-76c01ec35739\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-92ac8218-857db98746-dfhfs" Sep 09 10:45:27.122765 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:27.122745 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976b7a1c-81d9-41bc-9071-76c01ec35739-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-92ac8218-857db98746-dfhfs\" (UID: \"976b7a1c-81d9-41bc-9071-76c01ec35739\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-92ac8218-857db98746-dfhfs" Sep 09 10:45:27.132472 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:27.132447 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcsv9\" (UniqueName: \"kubernetes.io/projected/976b7a1c-81d9-41bc-9071-76c01ec35739-kube-api-access-bcsv9\") pod \"payload-pre-processing-e2e-migrate-92ac8218-857db98746-dfhfs\" (UID: \"976b7a1c-81d9-41bc-9071-76c01ec35739\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-92ac8218-857db98746-dfhfs" Sep 09 10:45:27.183997 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:27.183968 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-92ac8218-6bfbd47d8b-8vxvr" Sep 09 10:45:27.212014 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:27.211987 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-92ac8218-857db98746-dfhfs" Sep 09 10:45:27.474718 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:27.474692 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-92ac8218-6bfbd47d8b-8vxvr"] Sep 09 10:45:27.477474 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:45:27.477447 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56098277_d76a_4a15_945e_aff99da811cd.slice/crio-1b397e775823edcb8fa204a7cbcee70720f0afde6d390092660fb90b50a40f9e WatchSource:0}: Error finding container 1b397e775823edcb8fa204a7cbcee70720f0afde6d390092660fb90b50a40f9e: Status 404 returned error can't find the container with id 1b397e775823edcb8fa204a7cbcee70720f0afde6d390092660fb90b50a40f9e Sep 09 10:45:27.515561 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:27.515540 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-92ac8218-857db98746-dfhfs"] Sep 09 10:45:27.517580 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:45:27.517558 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod976b7a1c_81d9_41bc_9071_76c01ec35739.slice/crio-4524b598e7c6db3ff9debe25f99871b428c15da10daa8ac7c0e6091eaa8a1fc1 WatchSource:0}: Error finding container 4524b598e7c6db3ff9debe25f99871b428c15da10daa8ac7c0e6091eaa8a1fc1: Status 404 returned error can't find the container with id 4524b598e7c6db3ff9debe25f99871b428c15da10daa8ac7c0e6091eaa8a1fc1 Sep 09 10:45:28.030318 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:28.030270 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-92ac8218-6bfbd47d8b-8vxvr" event={"ID":"56098277-d76a-4a15-945e-aff99da811cd","Type":"ContainerStarted","Data":"133b69942efc96efde9ed1c175caf81d8006033cdafbf1efd86bfd7631de2e7d"} Sep 09 10:45:28.030790 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:28.030368 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-92ac8218-6bfbd47d8b-8vxvr" event={"ID":"56098277-d76a-4a15-945e-aff99da811cd","Type":"ContainerStarted","Data":"1b397e775823edcb8fa204a7cbcee70720f0afde6d390092660fb90b50a40f9e"} Sep 09 10:45:28.030790 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:28.030396 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-92ac8218-6bfbd47d8b-8vxvr" Sep 09 10:45:28.032987 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:28.032959 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-92ac8218-857db98746-dfhfs" event={"ID":"976b7a1c-81d9-41bc-9071-76c01ec35739","Type":"ContainerStarted","Data":"de44194b63f88d23a08832b3a9b06e97764efc08362563995e8c0b72d6b62518"} Sep 09 10:45:28.033141 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:28.033008 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-92ac8218-857db98746-dfhfs" event={"ID":"976b7a1c-81d9-41bc-9071-76c01ec35739","Type":"ContainerStarted","Data":"4524b598e7c6db3ff9debe25f99871b428c15da10daa8ac7c0e6091eaa8a1fc1"} Sep 09 10:45:28.033141 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:28.033057 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-92ac8218-857db98746-dfhfs" Sep 09 10:45:28.055032 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:28.054971 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-92ac8218-6bfbd47d8b-8vxvr" podStartSLOduration=2.054953132 podStartE2EDuration="2.054953132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:45:28.053544193 +0000 UTC m=+1168.228146518" watchObservedRunningTime="2026-09-09 10:45:28.054953132 +0000 UTC m=+1168.229555460" Sep 09 10:45:28.077554 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:28.077470 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-92ac8218-857db98746-dfhfs" podStartSLOduration=2.077453283 podStartE2EDuration="2.077453283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:45:28.0768751 +0000 UTC m=+1168.251477425" watchObservedRunningTime="2026-09-09 10:45:28.077453283 +0000 UTC m=+1168.252055608" Sep 09 10:45:39.041344 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:39.041305 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-92ac8218-6bfbd47d8b-8vxvr" Sep 09 10:45:39.041828 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:39.041492 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-92ac8218-857db98746-dfhfs" Sep 09 10:45:50.416458 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:50.416425 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6"] Sep 09 10:45:50.423764 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:50.423726 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" Sep 09 10:45:50.426667 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:50.426643 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-4bbfe3\"/\"default-dockercfg-tc9rk\"" Sep 09 10:45:50.427152 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:50.427129 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-4bbfe3\"/\"openshift-service-ca.crt\"" Sep 09 10:45:50.427828 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:50.427811 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-4bbfe3\"/\"autores-model-4bbfe3-kserve-self-signed-certs\"" Sep 09 10:45:50.429482 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:50.429462 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-4bbfe3\"/\"kube-root-ca.crt\"" Sep 09 10:45:50.434917 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:50.434886 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6"] Sep 09 10:45:50.525689 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:50.525650 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7drz\" (UniqueName: \"kubernetes.io/projected/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-kube-api-access-f7drz\") pod \"autores-model-4bbfe3-kserve-7c6876776c-r4pm6\" (UID: \"9b179542-2be1-4dad-9097-f3b4bc0d2dcb\") " pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" Sep 09 10:45:50.525863 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:50.525714 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-dshm\") pod \"autores-model-4bbfe3-kserve-7c6876776c-r4pm6\" (UID: \"9b179542-2be1-4dad-9097-f3b4bc0d2dcb\") " pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" Sep 09 10:45:50.525863 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:50.525732 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-tmp-dir\") pod \"autores-model-4bbfe3-kserve-7c6876776c-r4pm6\" (UID: \"9b179542-2be1-4dad-9097-f3b4bc0d2dcb\") " pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" Sep 09 10:45:50.525863 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:50.525750 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-model-cache\") pod \"autores-model-4bbfe3-kserve-7c6876776c-r4pm6\" (UID: \"9b179542-2be1-4dad-9097-f3b4bc0d2dcb\") " pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" Sep 09 10:45:50.525863 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:50.525797 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-home\") pod \"autores-model-4bbfe3-kserve-7c6876776c-r4pm6\" (UID: \"9b179542-2be1-4dad-9097-f3b4bc0d2dcb\") " pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" Sep 09 10:45:50.525863 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:50.525836 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-tls-certs\") pod \"autores-model-4bbfe3-kserve-7c6876776c-r4pm6\" (UID: \"9b179542-2be1-4dad-9097-f3b4bc0d2dcb\") " pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" Sep 09 10:45:50.627309 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:50.627271 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-tmp-dir\") pod \"autores-model-4bbfe3-kserve-7c6876776c-r4pm6\" (UID: \"9b179542-2be1-4dad-9097-f3b4bc0d2dcb\") " pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" Sep 09 10:45:50.627478 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:50.627317 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-model-cache\") pod \"autores-model-4bbfe3-kserve-7c6876776c-r4pm6\" (UID: \"9b179542-2be1-4dad-9097-f3b4bc0d2dcb\") " pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" Sep 09 10:45:50.627478 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:50.627441 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-home\") pod \"autores-model-4bbfe3-kserve-7c6876776c-r4pm6\" (UID: \"9b179542-2be1-4dad-9097-f3b4bc0d2dcb\") " pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" Sep 09 10:45:50.627595 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:50.627567 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-tls-certs\") pod \"autores-model-4bbfe3-kserve-7c6876776c-r4pm6\" (UID: \"9b179542-2be1-4dad-9097-f3b4bc0d2dcb\") " pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" Sep 09 10:45:50.627630 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:50.627616 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7drz\" (UniqueName: \"kubernetes.io/projected/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-kube-api-access-f7drz\") pod \"autores-model-4bbfe3-kserve-7c6876776c-r4pm6\" (UID: \"9b179542-2be1-4dad-9097-f3b4bc0d2dcb\") " pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" Sep 09 10:45:50.627694 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:50.627678 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-model-cache\") pod \"autores-model-4bbfe3-kserve-7c6876776c-r4pm6\" (UID: \"9b179542-2be1-4dad-9097-f3b4bc0d2dcb\") " pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" Sep 09 10:45:50.627747 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:50.627722 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-dshm\") pod \"autores-model-4bbfe3-kserve-7c6876776c-r4pm6\" (UID: \"9b179542-2be1-4dad-9097-f3b4bc0d2dcb\") " pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" Sep 09 10:45:50.627798 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:50.627749 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-tmp-dir\") pod \"autores-model-4bbfe3-kserve-7c6876776c-r4pm6\" (UID: \"9b179542-2be1-4dad-9097-f3b4bc0d2dcb\") " pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" Sep 09 10:45:50.627846 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:50.627822 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-home\") pod \"autores-model-4bbfe3-kserve-7c6876776c-r4pm6\" (UID: \"9b179542-2be1-4dad-9097-f3b4bc0d2dcb\") " pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" Sep 09 10:45:50.629821 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:50.629799 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-dshm\") pod \"autores-model-4bbfe3-kserve-7c6876776c-r4pm6\" (UID: \"9b179542-2be1-4dad-9097-f3b4bc0d2dcb\") " pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" Sep 09 10:45:50.630162 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:50.630143 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-tls-certs\") pod \"autores-model-4bbfe3-kserve-7c6876776c-r4pm6\" (UID: \"9b179542-2be1-4dad-9097-f3b4bc0d2dcb\") " pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" Sep 09 10:45:50.639343 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:50.639313 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7drz\" (UniqueName: \"kubernetes.io/projected/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-kube-api-access-f7drz\") pod \"autores-model-4bbfe3-kserve-7c6876776c-r4pm6\" (UID: \"9b179542-2be1-4dad-9097-f3b4bc0d2dcb\") " pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" Sep 09 10:45:50.733482 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:50.733400 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" Sep 09 10:45:50.912439 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:50.912404 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ac8218-bmt96"] Sep 09 10:45:50.916203 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:50.916179 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ac8218-bmt96" Sep 09 10:45:50.919166 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:50.919129 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 10:45:50.919166 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:50.919159 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kq7qv\"" Sep 09 10:45:50.925713 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:50.925685 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ac8218-bmt96"] Sep 09 10:45:51.008743 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:51.008652 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6"] Sep 09 10:45:51.030919 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:45:51.030892 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b179542_2be1_4dad_9097_f3b4bc0d2dcb.slice/crio-e0053a02c77802bb2f1bfda2c6f414ad4039f87a716a65f890b1fc9da376c025 WatchSource:0}: Error finding container e0053a02c77802bb2f1bfda2c6f414ad4039f87a716a65f890b1fc9da376c025: Status 404 returned error can't find the container with id e0053a02c77802bb2f1bfda2c6f414ad4039f87a716a65f890b1fc9da376c025 Sep 09 10:45:51.031056 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:51.031029 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43199450-ac9a-4e8d-a7cf-7ece9178d18d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-92ac8218-bmt96\" (UID: \"43199450-ac9a-4e8d-a7cf-7ece9178d18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ac8218-bmt96" Sep 09 10:45:51.132578 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:51.132541 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43199450-ac9a-4e8d-a7cf-7ece9178d18d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-92ac8218-bmt96\" (UID: \"43199450-ac9a-4e8d-a7cf-7ece9178d18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ac8218-bmt96" Sep 09 10:45:51.133181 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:51.133159 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43199450-ac9a-4e8d-a7cf-7ece9178d18d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-92ac8218-bmt96\" (UID: \"43199450-ac9a-4e8d-a7cf-7ece9178d18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ac8218-bmt96" Sep 09 10:45:51.154131 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:51.154093 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" event={"ID":"9b179542-2be1-4dad-9097-f3b4bc0d2dcb","Type":"ContainerStarted","Data":"dc642b7be06a53ecd50ac4d45f1118f6e9421d7c19fab01040d09e870c5df1fc"} Sep 09 10:45:51.154293 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:51.154141 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" event={"ID":"9b179542-2be1-4dad-9097-f3b4bc0d2dcb","Type":"ContainerStarted","Data":"e0053a02c77802bb2f1bfda2c6f414ad4039f87a716a65f890b1fc9da376c025"} Sep 09 10:45:51.173183 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:51.173107 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" podStartSLOduration=1.17308501 podStartE2EDuration="1.17308501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:45:51.172945653 +0000 UTC m=+1191.347547979" watchObservedRunningTime="2026-09-09 10:45:51.17308501 +0000 UTC m=+1191.347687336" Sep 09 10:45:51.227815 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:51.227772 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ac8218-bmt96" Sep 09 10:45:51.503347 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:51.503322 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ac8218-bmt96"] Sep 09 10:45:51.505855 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:45:51.505827 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43199450_ac9a_4e8d_a7cf_7ece9178d18d.slice/crio-be2639901cd090247abcd9398b104dbeb53d01fc82bdc369351fe802d7ee77e0 WatchSource:0}: Error finding container be2639901cd090247abcd9398b104dbeb53d01fc82bdc369351fe802d7ee77e0: Status 404 returned error can't find the container with id be2639901cd090247abcd9398b104dbeb53d01fc82bdc369351fe802d7ee77e0 Sep 09 10:45:52.161233 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:52.161196 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ac8218-bmt96" event={"ID":"43199450-ac9a-4e8d-a7cf-7ece9178d18d","Type":"ContainerStarted","Data":"166af4109bd0d3e0fcc05ff2520448b1a710ed93f3473e288591c4e32ffcc6ee"} Sep 09 10:45:52.161420 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:52.161257 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ac8218-bmt96" event={"ID":"43199450-ac9a-4e8d-a7cf-7ece9178d18d","Type":"ContainerStarted","Data":"be2639901cd090247abcd9398b104dbeb53d01fc82bdc369351fe802d7ee77e0"} Sep 09 10:45:52.182587 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:52.182529 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ac8218-bmt96" podStartSLOduration=2.182489149 podStartE2EDuration="2.182489149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:45:52.180435369 +0000 UTC m=+1192.355037693" watchObservedRunningTime="2026-09-09 10:45:52.182489149 +0000 UTC m=+1192.357091473" Sep 09 10:45:53.168380 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:53.168340 2559 generic.go:356] "Generic (PLEG): container finished" podID="43199450-ac9a-4e8d-a7cf-7ece9178d18d" containerID="166af4109bd0d3e0fcc05ff2520448b1a710ed93f3473e288591c4e32ffcc6ee" exitCode=0 Sep 09 10:45:53.168827 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:53.168431 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ac8218-bmt96" event={"ID":"43199450-ac9a-4e8d-a7cf-7ece9178d18d","Type":"ContainerDied","Data":"166af4109bd0d3e0fcc05ff2520448b1a710ed93f3473e288591c4e32ffcc6ee"} Sep 09 10:45:54.314612 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:54.314579 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ac8218-bmt96" Sep 09 10:45:54.465411 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:54.465306 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43199450-ac9a-4e8d-a7cf-7ece9178d18d-maas-api-service-ca\") pod \"43199450-ac9a-4e8d-a7cf-7ece9178d18d\" (UID: \"43199450-ac9a-4e8d-a7cf-7ece9178d18d\") " Sep 09 10:45:54.465726 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:54.465699 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43199450-ac9a-4e8d-a7cf-7ece9178d18d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "43199450-ac9a-4e8d-a7cf-7ece9178d18d" (UID: "43199450-ac9a-4e8d-a7cf-7ece9178d18d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:45:54.566945 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:54.566912 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43199450-ac9a-4e8d-a7cf-7ece9178d18d-maas-api-service-ca\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:45:55.179949 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:55.179914 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ac8218-bmt96" Sep 09 10:45:55.179949 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:55.179925 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ac8218-bmt96" event={"ID":"43199450-ac9a-4e8d-a7cf-7ece9178d18d","Type":"ContainerDied","Data":"be2639901cd090247abcd9398b104dbeb53d01fc82bdc369351fe802d7ee77e0"} Sep 09 10:45:55.180162 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:55.179970 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be2639901cd090247abcd9398b104dbeb53d01fc82bdc369351fe802d7ee77e0" Sep 09 10:45:57.636517 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:57.636467 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-92ac8218-6bfbd47d8b-8vxvr"] Sep 09 10:45:57.637001 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:57.636758 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-92ac8218-6bfbd47d8b-8vxvr" podUID="56098277-d76a-4a15-945e-aff99da811cd" containerName="payload-processing" containerID="cri-o://133b69942efc96efde9ed1c175caf81d8006033cdafbf1efd86bfd7631de2e7d" gracePeriod=30 Sep 09 10:45:57.648670 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:57.644461 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-92ac8218-857db98746-dfhfs"] Sep 09 10:45:57.648670 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:57.647144 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-92ac8218-857db98746-dfhfs" podUID="976b7a1c-81d9-41bc-9071-76c01ec35739" containerName="payload-pre-processing" containerID="cri-o://de44194b63f88d23a08832b3a9b06e97764efc08362563995e8c0b72d6b62518" gracePeriod=30 Sep 09 10:45:57.821971 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:57.821844 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ac8218-bmt96"] Sep 09 10:45:57.825677 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:57.825640 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-92ac8218-bmt96"] Sep 09 10:45:57.900055 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:57.899987 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-92ac8218-6bfbd47d8b-8vxvr" Sep 09 10:45:57.946006 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:57.945974 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-92ac8218-857db98746-dfhfs" Sep 09 10:45:57.996392 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:57.996354 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s69s9\" (UniqueName: \"kubernetes.io/projected/56098277-d76a-4a15-945e-aff99da811cd-kube-api-access-s69s9\") pod \"56098277-d76a-4a15-945e-aff99da811cd\" (UID: \"56098277-d76a-4a15-945e-aff99da811cd\") " Sep 09 10:45:57.996625 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:57.996445 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56098277-d76a-4a15-945e-aff99da811cd-plugins-config-volume\") pod \"56098277-d76a-4a15-945e-aff99da811cd\" (UID: \"56098277-d76a-4a15-945e-aff99da811cd\") " Sep 09 10:45:57.996820 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:57.996798 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56098277-d76a-4a15-945e-aff99da811cd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "56098277-d76a-4a15-945e-aff99da811cd" (UID: "56098277-d76a-4a15-945e-aff99da811cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:45:57.998480 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:57.998457 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56098277-d76a-4a15-945e-aff99da811cd-kube-api-access-s69s9" (OuterVolumeSpecName: "kube-api-access-s69s9") pod "56098277-d76a-4a15-945e-aff99da811cd" (UID: "56098277-d76a-4a15-945e-aff99da811cd"). InnerVolumeSpecName "kube-api-access-s69s9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:45:58.097956 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:58.097910 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcsv9\" (UniqueName: \"kubernetes.io/projected/976b7a1c-81d9-41bc-9071-76c01ec35739-kube-api-access-bcsv9\") pod \"976b7a1c-81d9-41bc-9071-76c01ec35739\" (UID: \"976b7a1c-81d9-41bc-9071-76c01ec35739\") " Sep 09 10:45:58.098153 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:58.097987 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976b7a1c-81d9-41bc-9071-76c01ec35739-plugins-config-volume\") pod \"976b7a1c-81d9-41bc-9071-76c01ec35739\" (UID: \"976b7a1c-81d9-41bc-9071-76c01ec35739\") " Sep 09 10:45:58.098280 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:58.098260 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56098277-d76a-4a15-945e-aff99da811cd-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:45:58.098341 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:58.098286 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s69s9\" (UniqueName: \"kubernetes.io/projected/56098277-d76a-4a15-945e-aff99da811cd-kube-api-access-s69s9\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:45:58.098414 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:58.098382 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/976b7a1c-81d9-41bc-9071-76c01ec35739-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "976b7a1c-81d9-41bc-9071-76c01ec35739" (UID: "976b7a1c-81d9-41bc-9071-76c01ec35739"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:45:58.100046 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:58.100023 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/976b7a1c-81d9-41bc-9071-76c01ec35739-kube-api-access-bcsv9" (OuterVolumeSpecName: "kube-api-access-bcsv9") pod "976b7a1c-81d9-41bc-9071-76c01ec35739" (UID: "976b7a1c-81d9-41bc-9071-76c01ec35739"). InnerVolumeSpecName "kube-api-access-bcsv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:45:58.197141 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:58.197057 2559 generic.go:356] "Generic (PLEG): container finished" podID="56098277-d76a-4a15-945e-aff99da811cd" containerID="133b69942efc96efde9ed1c175caf81d8006033cdafbf1efd86bfd7631de2e7d" exitCode=0 Sep 09 10:45:58.197141 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:58.197108 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-92ac8218-6bfbd47d8b-8vxvr" Sep 09 10:45:58.197360 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:58.197136 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-92ac8218-6bfbd47d8b-8vxvr" event={"ID":"56098277-d76a-4a15-945e-aff99da811cd","Type":"ContainerDied","Data":"133b69942efc96efde9ed1c175caf81d8006033cdafbf1efd86bfd7631de2e7d"} Sep 09 10:45:58.197360 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:58.197193 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-92ac8218-6bfbd47d8b-8vxvr" event={"ID":"56098277-d76a-4a15-945e-aff99da811cd","Type":"ContainerDied","Data":"1b397e775823edcb8fa204a7cbcee70720f0afde6d390092660fb90b50a40f9e"} Sep 09 10:45:58.197360 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:58.197226 2559 scope.go:117] "RemoveContainer" containerID="133b69942efc96efde9ed1c175caf81d8006033cdafbf1efd86bfd7631de2e7d" Sep 09 10:45:58.198595 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:58.198572 2559 generic.go:356] "Generic (PLEG): container finished" podID="976b7a1c-81d9-41bc-9071-76c01ec35739" containerID="de44194b63f88d23a08832b3a9b06e97764efc08362563995e8c0b72d6b62518" exitCode=0 Sep 09 10:45:58.198703 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:58.198620 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-92ac8218-857db98746-dfhfs" Sep 09 10:45:58.198703 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:58.198663 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-92ac8218-857db98746-dfhfs" event={"ID":"976b7a1c-81d9-41bc-9071-76c01ec35739","Type":"ContainerDied","Data":"de44194b63f88d23a08832b3a9b06e97764efc08362563995e8c0b72d6b62518"} Sep 09 10:45:58.198805 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:58.198707 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-92ac8218-857db98746-dfhfs" event={"ID":"976b7a1c-81d9-41bc-9071-76c01ec35739","Type":"ContainerDied","Data":"4524b598e7c6db3ff9debe25f99871b428c15da10daa8ac7c0e6091eaa8a1fc1"} Sep 09 10:45:58.198805 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:58.198739 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcsv9\" (UniqueName: \"kubernetes.io/projected/976b7a1c-81d9-41bc-9071-76c01ec35739-kube-api-access-bcsv9\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:45:58.198805 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:58.198778 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976b7a1c-81d9-41bc-9071-76c01ec35739-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:45:58.210725 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:58.207818 2559 scope.go:117] "RemoveContainer" containerID="133b69942efc96efde9ed1c175caf81d8006033cdafbf1efd86bfd7631de2e7d" Sep 09 10:45:58.210725 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:45:58.210619 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"133b69942efc96efde9ed1c175caf81d8006033cdafbf1efd86bfd7631de2e7d\": container with ID starting with 133b69942efc96efde9ed1c175caf81d8006033cdafbf1efd86bfd7631de2e7d not found: ID does not exist" containerID="133b69942efc96efde9ed1c175caf81d8006033cdafbf1efd86bfd7631de2e7d" Sep 09 10:45:58.210725 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:58.210665 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"133b69942efc96efde9ed1c175caf81d8006033cdafbf1efd86bfd7631de2e7d"} err="failed to get container status \"133b69942efc96efde9ed1c175caf81d8006033cdafbf1efd86bfd7631de2e7d\": rpc error: code = NotFound desc = could not find container \"133b69942efc96efde9ed1c175caf81d8006033cdafbf1efd86bfd7631de2e7d\": container with ID starting with 133b69942efc96efde9ed1c175caf81d8006033cdafbf1efd86bfd7631de2e7d not found: ID does not exist" Sep 09 10:45:58.210725 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:58.210701 2559 scope.go:117] "RemoveContainer" containerID="de44194b63f88d23a08832b3a9b06e97764efc08362563995e8c0b72d6b62518" Sep 09 10:45:58.219837 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:58.219817 2559 scope.go:117] "RemoveContainer" containerID="de44194b63f88d23a08832b3a9b06e97764efc08362563995e8c0b72d6b62518" Sep 09 10:45:58.220142 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:45:58.220122 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de44194b63f88d23a08832b3a9b06e97764efc08362563995e8c0b72d6b62518\": container with ID starting with de44194b63f88d23a08832b3a9b06e97764efc08362563995e8c0b72d6b62518 not found: ID does not exist" containerID="de44194b63f88d23a08832b3a9b06e97764efc08362563995e8c0b72d6b62518" Sep 09 10:45:58.220181 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:58.220153 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de44194b63f88d23a08832b3a9b06e97764efc08362563995e8c0b72d6b62518"} err="failed to get container status \"de44194b63f88d23a08832b3a9b06e97764efc08362563995e8c0b72d6b62518\": rpc error: code = NotFound desc = could not find container \"de44194b63f88d23a08832b3a9b06e97764efc08362563995e8c0b72d6b62518\": container with ID starting with de44194b63f88d23a08832b3a9b06e97764efc08362563995e8c0b72d6b62518 not found: ID does not exist" Sep 09 10:45:58.228007 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:58.227980 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-92ac8218-6bfbd47d8b-8vxvr"] Sep 09 10:45:58.232531 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:58.232489 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-92ac8218-6bfbd47d8b-8vxvr"] Sep 09 10:45:58.244617 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:58.244578 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-92ac8218-857db98746-dfhfs"] Sep 09 10:45:58.250108 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:58.250081 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-92ac8218-857db98746-dfhfs"] Sep 09 10:45:58.334754 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:58.334720 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43199450-ac9a-4e8d-a7cf-7ece9178d18d" path="/var/lib/kubelet/pods/43199450-ac9a-4e8d-a7cf-7ece9178d18d/volumes" Sep 09 10:45:58.335079 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:58.335066 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56098277-d76a-4a15-945e-aff99da811cd" path="/var/lib/kubelet/pods/56098277-d76a-4a15-945e-aff99da811cd/volumes" Sep 09 10:45:58.335378 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:45:58.335366 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="976b7a1c-81d9-41bc-9071-76c01ec35739" path="/var/lib/kubelet/pods/976b7a1c-81d9-41bc-9071-76c01ec35739/volumes" Sep 09 10:46:00.303677 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:00.303640 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d22ff_0a56b285-3f3c-4f6d-a939-8a95ae00e8da/ovn-acl-logging/0.log" Sep 09 10:46:00.306537 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:00.306490 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d22ff_0a56b285-3f3c-4f6d-a939-8a95ae00e8da/ovn-acl-logging/0.log" Sep 09 10:46:00.734574 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:00.734535 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" Sep 09 10:46:00.734770 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:00.734594 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" Sep 09 10:46:00.747166 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:00.747138 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" Sep 09 10:46:01.227488 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:01.227454 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" Sep 09 10:46:02.911228 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:02.911189 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dfb9af4b-6465749495-nx28m"] Sep 09 10:46:02.911822 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:02.911797 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43199450-ac9a-4e8d-a7cf-7ece9178d18d" containerName="revoke-keys" Sep 09 10:46:02.911822 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:02.911820 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="43199450-ac9a-4e8d-a7cf-7ece9178d18d" containerName="revoke-keys" Sep 09 10:46:02.911938 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:02.911874 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="976b7a1c-81d9-41bc-9071-76c01ec35739" containerName="payload-pre-processing" Sep 09 10:46:02.911938 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:02.911883 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="976b7a1c-81d9-41bc-9071-76c01ec35739" containerName="payload-pre-processing" Sep 09 10:46:02.911938 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:02.911897 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56098277-d76a-4a15-945e-aff99da811cd" containerName="payload-processing" Sep 09 10:46:02.911938 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:02.911907 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="56098277-d76a-4a15-945e-aff99da811cd" containerName="payload-processing" Sep 09 10:46:02.912137 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:02.912005 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="56098277-d76a-4a15-945e-aff99da811cd" containerName="payload-processing" Sep 09 10:46:02.912137 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:02.912022 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="976b7a1c-81d9-41bc-9071-76c01ec35739" containerName="payload-pre-processing" Sep 09 10:46:02.912137 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:02.912034 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="43199450-ac9a-4e8d-a7cf-7ece9178d18d" containerName="revoke-keys" Sep 09 10:46:02.916264 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:02.916242 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dfb9af4b-6465749495-nx28m" Sep 09 10:46:02.919317 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:02.919296 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-dfb9af4b\"" Sep 09 10:46:02.919473 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:02.919457 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-dfb9af4b-dockercfg-8tg9l\"" Sep 09 10:46:02.929411 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:02.929384 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dfb9af4b-6465749495-nx28m"] Sep 09 10:46:02.979181 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:02.979147 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dfb9af4b-694f749f5d-qnmpf"] Sep 09 10:46:02.982470 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:02.982441 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dfb9af4b-694f749f5d-qnmpf" Sep 09 10:46:02.997554 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:02.997527 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dfb9af4b-694f749f5d-qnmpf"] Sep 09 10:46:03.042817 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:03.042773 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7bg2\" (UniqueName: \"kubernetes.io/projected/c1deb148-0a7b-454d-b2bd-fec2426dbbca-kube-api-access-b7bg2\") pod \"payload-processing-e2e-ait-dfb9af4b-6465749495-nx28m\" (UID: \"c1deb148-0a7b-454d-b2bd-fec2426dbbca\") " pod="openshift-ingress/payload-processing-e2e-ait-dfb9af4b-6465749495-nx28m" Sep 09 10:46:03.043095 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:03.043073 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1deb148-0a7b-454d-b2bd-fec2426dbbca-plugins-config-volume\") pod \"payload-processing-e2e-ait-dfb9af4b-6465749495-nx28m\" (UID: \"c1deb148-0a7b-454d-b2bd-fec2426dbbca\") " pod="openshift-ingress/payload-processing-e2e-ait-dfb9af4b-6465749495-nx28m" Sep 09 10:46:03.144530 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:03.144448 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7bg2\" (UniqueName: \"kubernetes.io/projected/c1deb148-0a7b-454d-b2bd-fec2426dbbca-kube-api-access-b7bg2\") pod \"payload-processing-e2e-ait-dfb9af4b-6465749495-nx28m\" (UID: \"c1deb148-0a7b-454d-b2bd-fec2426dbbca\") " pod="openshift-ingress/payload-processing-e2e-ait-dfb9af4b-6465749495-nx28m" Sep 09 10:46:03.144530 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:03.144534 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1deb148-0a7b-454d-b2bd-fec2426dbbca-plugins-config-volume\") pod \"payload-processing-e2e-ait-dfb9af4b-6465749495-nx28m\" (UID: \"c1deb148-0a7b-454d-b2bd-fec2426dbbca\") " pod="openshift-ingress/payload-processing-e2e-ait-dfb9af4b-6465749495-nx28m" Sep 09 10:46:03.144750 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:03.144566 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7t5b\" (UniqueName: \"kubernetes.io/projected/11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9-kube-api-access-k7t5b\") pod \"payload-pre-processing-e2e-ait-dfb9af4b-694f749f5d-qnmpf\" (UID: \"11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dfb9af4b-694f749f5d-qnmpf" Sep 09 10:46:03.144750 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:03.144665 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dfb9af4b-694f749f5d-qnmpf\" (UID: \"11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dfb9af4b-694f749f5d-qnmpf" Sep 09 10:46:03.145405 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:03.145386 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1deb148-0a7b-454d-b2bd-fec2426dbbca-plugins-config-volume\") pod \"payload-processing-e2e-ait-dfb9af4b-6465749495-nx28m\" (UID: \"c1deb148-0a7b-454d-b2bd-fec2426dbbca\") " pod="openshift-ingress/payload-processing-e2e-ait-dfb9af4b-6465749495-nx28m" Sep 09 10:46:03.161702 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:03.161622 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7bg2\" (UniqueName: \"kubernetes.io/projected/c1deb148-0a7b-454d-b2bd-fec2426dbbca-kube-api-access-b7bg2\") pod \"payload-processing-e2e-ait-dfb9af4b-6465749495-nx28m\" (UID: \"c1deb148-0a7b-454d-b2bd-fec2426dbbca\") " pod="openshift-ingress/payload-processing-e2e-ait-dfb9af4b-6465749495-nx28m" Sep 09 10:46:03.226867 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:03.226836 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dfb9af4b-6465749495-nx28m" Sep 09 10:46:03.245462 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:03.245425 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7t5b\" (UniqueName: \"kubernetes.io/projected/11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9-kube-api-access-k7t5b\") pod \"payload-pre-processing-e2e-ait-dfb9af4b-694f749f5d-qnmpf\" (UID: \"11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dfb9af4b-694f749f5d-qnmpf" Sep 09 10:46:03.245774 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:03.245750 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dfb9af4b-694f749f5d-qnmpf\" (UID: \"11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dfb9af4b-694f749f5d-qnmpf" Sep 09 10:46:03.246857 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:03.246820 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dfb9af4b-694f749f5d-qnmpf\" (UID: \"11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dfb9af4b-694f749f5d-qnmpf" Sep 09 10:46:03.257080 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:03.257051 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7t5b\" (UniqueName: \"kubernetes.io/projected/11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9-kube-api-access-k7t5b\") pod \"payload-pre-processing-e2e-ait-dfb9af4b-694f749f5d-qnmpf\" (UID: \"11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dfb9af4b-694f749f5d-qnmpf" Sep 09 10:46:03.293886 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:03.293854 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dfb9af4b-694f749f5d-qnmpf" Sep 09 10:46:03.509482 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:03.509456 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dfb9af4b-6465749495-nx28m"] Sep 09 10:46:03.511900 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:46:03.511872 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1deb148_0a7b_454d_b2bd_fec2426dbbca.slice/crio-4adc8b38d7459d724e4a91dc5ef4aa97a41e94f22f6c097b6bc2696b84107dac WatchSource:0}: Error finding container 4adc8b38d7459d724e4a91dc5ef4aa97a41e94f22f6c097b6bc2696b84107dac: Status 404 returned error can't find the container with id 4adc8b38d7459d724e4a91dc5ef4aa97a41e94f22f6c097b6bc2696b84107dac Sep 09 10:46:03.575136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:03.575102 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dfb9af4b-694f749f5d-qnmpf"] Sep 09 10:46:03.578062 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:46:03.578033 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11fa8a5c_ea9d_49c4_9a7c_25cbbc86d8a9.slice/crio-45cf0baa167653ede1134f8fd42a47c4835ab903b31a299bde0e7061269ed7a9 WatchSource:0}: Error finding container 45cf0baa167653ede1134f8fd42a47c4835ab903b31a299bde0e7061269ed7a9: Status 404 returned error can't find the container with id 45cf0baa167653ede1134f8fd42a47c4835ab903b31a299bde0e7061269ed7a9 Sep 09 10:46:04.232392 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:04.232352 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dfb9af4b-6465749495-nx28m" event={"ID":"c1deb148-0a7b-454d-b2bd-fec2426dbbca","Type":"ContainerStarted","Data":"fb3c0efa8a07df37b5948971b5948e4db219ac16adc308be995fe2943fa881c7"} Sep 09 10:46:04.232891 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:04.232406 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dfb9af4b-6465749495-nx28m" event={"ID":"c1deb148-0a7b-454d-b2bd-fec2426dbbca","Type":"ContainerStarted","Data":"4adc8b38d7459d724e4a91dc5ef4aa97a41e94f22f6c097b6bc2696b84107dac"} Sep 09 10:46:04.232891 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:04.232541 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-dfb9af4b-6465749495-nx28m" Sep 09 10:46:04.233994 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:04.233970 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dfb9af4b-694f749f5d-qnmpf" event={"ID":"11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9","Type":"ContainerStarted","Data":"7141409489e10c55cba2b5e0827189c20e111298ed263d4a5b2c9962d89a5db3"} Sep 09 10:46:04.234103 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:04.234016 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-dfb9af4b-694f749f5d-qnmpf" Sep 09 10:46:04.234103 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:04.234032 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dfb9af4b-694f749f5d-qnmpf" event={"ID":"11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9","Type":"ContainerStarted","Data":"45cf0baa167653ede1134f8fd42a47c4835ab903b31a299bde0e7061269ed7a9"} Sep 09 10:46:04.251461 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:04.251405 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-dfb9af4b-6465749495-nx28m" podStartSLOduration=2.251383401 podStartE2EDuration="2.251383401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:46:04.250383566 +0000 UTC m=+1204.424985892" watchObservedRunningTime="2026-09-09 10:46:04.251383401 +0000 UTC m=+1204.425985727" Sep 09 10:46:04.269407 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:04.269338 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-dfb9af4b-694f749f5d-qnmpf" podStartSLOduration=2.269322126 podStartE2EDuration="2.269322126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:46:04.267134097 +0000 UTC m=+1204.441736422" watchObservedRunningTime="2026-09-09 10:46:04.269322126 +0000 UTC m=+1204.443924450" Sep 09 10:46:15.241603 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:15.241547 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-dfb9af4b-6465749495-nx28m" Sep 09 10:46:15.242044 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:15.241679 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-dfb9af4b-694f749f5d-qnmpf" Sep 09 10:46:24.120716 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:24.120677 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4bbfe3-fstc7"] Sep 09 10:46:24.124155 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:24.124139 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4bbfe3-fstc7" Sep 09 10:46:24.126805 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:24.126778 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 10:46:24.126926 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:24.126778 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kq7qv\"" Sep 09 10:46:24.132424 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:24.132394 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4bbfe3-fstc7"] Sep 09 10:46:24.229846 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:24.229807 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0b60b4c-8233-4918-b268-e544239894c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-4bbfe3-fstc7\" (UID: \"e0b60b4c-8233-4918-b268-e544239894c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4bbfe3-fstc7" Sep 09 10:46:24.330634 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:24.330596 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0b60b4c-8233-4918-b268-e544239894c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-4bbfe3-fstc7\" (UID: \"e0b60b4c-8233-4918-b268-e544239894c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4bbfe3-fstc7" Sep 09 10:46:24.331270 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:24.331251 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0b60b4c-8233-4918-b268-e544239894c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-4bbfe3-fstc7\" (UID: \"e0b60b4c-8233-4918-b268-e544239894c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4bbfe3-fstc7" Sep 09 10:46:24.436545 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:24.436449 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4bbfe3-fstc7" Sep 09 10:46:24.708979 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:24.708955 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4bbfe3-fstc7"] Sep 09 10:46:24.711603 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:46:24.711573 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b60b4c_8233_4918_b268_e544239894c4.slice/crio-6261f1a02aa6c8ee3fb708eec8e086834c9864f9470ea73c17d3519846225762 WatchSource:0}: Error finding container 6261f1a02aa6c8ee3fb708eec8e086834c9864f9470ea73c17d3519846225762: Status 404 returned error can't find the container with id 6261f1a02aa6c8ee3fb708eec8e086834c9864f9470ea73c17d3519846225762 Sep 09 10:46:25.352360 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:25.352323 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4bbfe3-fstc7" event={"ID":"e0b60b4c-8233-4918-b268-e544239894c4","Type":"ContainerStarted","Data":"d6290ded53fcddc43798d310e3fd72a0198df7dc6a8c3da44d766f25cba86174"} Sep 09 10:46:25.352768 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:25.352398 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4bbfe3-fstc7" event={"ID":"e0b60b4c-8233-4918-b268-e544239894c4","Type":"ContainerStarted","Data":"6261f1a02aa6c8ee3fb708eec8e086834c9864f9470ea73c17d3519846225762"} Sep 09 10:46:25.373411 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:25.373357 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4bbfe3-fstc7" podStartSLOduration=1.373339597 podStartE2EDuration="1.373339597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:46:25.372110822 +0000 UTC m=+1225.546713148" watchObservedRunningTime="2026-09-09 10:46:25.373339597 +0000 UTC m=+1225.547941921" Sep 09 10:46:26.358033 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:26.357994 2559 generic.go:356] "Generic (PLEG): container finished" podID="e0b60b4c-8233-4918-b268-e544239894c4" containerID="d6290ded53fcddc43798d310e3fd72a0198df7dc6a8c3da44d766f25cba86174" exitCode=0 Sep 09 10:46:26.358411 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:26.358042 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4bbfe3-fstc7" event={"ID":"e0b60b4c-8233-4918-b268-e544239894c4","Type":"ContainerDied","Data":"d6290ded53fcddc43798d310e3fd72a0198df7dc6a8c3da44d766f25cba86174"} Sep 09 10:46:27.497291 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:27.497266 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4bbfe3-fstc7" Sep 09 10:46:27.558942 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:27.558911 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0b60b4c-8233-4918-b268-e544239894c4-maas-api-service-ca\") pod \"e0b60b4c-8233-4918-b268-e544239894c4\" (UID: \"e0b60b4c-8233-4918-b268-e544239894c4\") " Sep 09 10:46:27.559310 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:27.559279 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0b60b4c-8233-4918-b268-e544239894c4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e0b60b4c-8233-4918-b268-e544239894c4" (UID: "e0b60b4c-8233-4918-b268-e544239894c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:46:27.659640 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:27.659554 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0b60b4c-8233-4918-b268-e544239894c4-maas-api-service-ca\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:46:28.369578 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:28.369549 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4bbfe3-fstc7" Sep 09 10:46:28.369578 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:28.369562 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4bbfe3-fstc7" event={"ID":"e0b60b4c-8233-4918-b268-e544239894c4","Type":"ContainerDied","Data":"6261f1a02aa6c8ee3fb708eec8e086834c9864f9470ea73c17d3519846225762"} Sep 09 10:46:28.369752 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:28.369609 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6261f1a02aa6c8ee3fb708eec8e086834c9864f9470ea73c17d3519846225762" Sep 09 10:46:32.072019 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:32.071982 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6"] Sep 09 10:46:33.132444 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:33.132408 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dfb9af4b-6465749495-nx28m"] Sep 09 10:46:33.132937 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:33.132626 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-dfb9af4b-6465749495-nx28m" podUID="c1deb148-0a7b-454d-b2bd-fec2426dbbca" containerName="payload-processing" containerID="cri-o://fb3c0efa8a07df37b5948971b5948e4db219ac16adc308be995fe2943fa881c7" gracePeriod=30 Sep 09 10:46:33.147488 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:33.147453 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dfb9af4b-694f749f5d-qnmpf"] Sep 09 10:46:33.147964 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:33.147911 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-dfb9af4b-694f749f5d-qnmpf" podUID="11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9" containerName="payload-pre-processing" containerID="cri-o://7141409489e10c55cba2b5e0827189c20e111298ed263d4a5b2c9962d89a5db3" gracePeriod=30 Sep 09 10:46:33.401302 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:33.401223 2559 generic.go:356] "Generic (PLEG): container finished" podID="11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9" containerID="7141409489e10c55cba2b5e0827189c20e111298ed263d4a5b2c9962d89a5db3" exitCode=0 Sep 09 10:46:33.401471 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:33.401306 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dfb9af4b-694f749f5d-qnmpf" event={"ID":"11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9","Type":"ContainerDied","Data":"7141409489e10c55cba2b5e0827189c20e111298ed263d4a5b2c9962d89a5db3"} Sep 09 10:46:33.402933 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:33.402906 2559 generic.go:356] "Generic (PLEG): container finished" podID="c1deb148-0a7b-454d-b2bd-fec2426dbbca" containerID="fb3c0efa8a07df37b5948971b5948e4db219ac16adc308be995fe2943fa881c7" exitCode=0 Sep 09 10:46:33.403036 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:33.402986 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dfb9af4b-6465749495-nx28m" event={"ID":"c1deb148-0a7b-454d-b2bd-fec2426dbbca","Type":"ContainerDied","Data":"fb3c0efa8a07df37b5948971b5948e4db219ac16adc308be995fe2943fa881c7"} Sep 09 10:46:33.429947 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:33.429912 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dfb9af4b-6465749495-nx28m" Sep 09 10:46:33.433898 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:33.433747 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dfb9af4b-694f749f5d-qnmpf" Sep 09 10:46:33.511684 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:33.511647 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7bg2\" (UniqueName: \"kubernetes.io/projected/c1deb148-0a7b-454d-b2bd-fec2426dbbca-kube-api-access-b7bg2\") pod \"c1deb148-0a7b-454d-b2bd-fec2426dbbca\" (UID: \"c1deb148-0a7b-454d-b2bd-fec2426dbbca\") " Sep 09 10:46:33.511875 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:33.511696 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7t5b\" (UniqueName: \"kubernetes.io/projected/11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9-kube-api-access-k7t5b\") pod \"11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9\" (UID: \"11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9\") " Sep 09 10:46:33.511875 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:33.511725 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9-plugins-config-volume\") pod \"11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9\" (UID: \"11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9\") " Sep 09 10:46:33.511875 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:33.511775 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1deb148-0a7b-454d-b2bd-fec2426dbbca-plugins-config-volume\") pod \"c1deb148-0a7b-454d-b2bd-fec2426dbbca\" (UID: \"c1deb148-0a7b-454d-b2bd-fec2426dbbca\") " Sep 09 10:46:33.512164 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:33.512138 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9" (UID: "11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:46:33.512239 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:33.512174 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1deb148-0a7b-454d-b2bd-fec2426dbbca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c1deb148-0a7b-454d-b2bd-fec2426dbbca" (UID: "c1deb148-0a7b-454d-b2bd-fec2426dbbca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:46:33.513869 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:33.513845 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9-kube-api-access-k7t5b" (OuterVolumeSpecName: "kube-api-access-k7t5b") pod "11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9" (UID: "11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9"). InnerVolumeSpecName "kube-api-access-k7t5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:46:33.513933 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:33.513892 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1deb148-0a7b-454d-b2bd-fec2426dbbca-kube-api-access-b7bg2" (OuterVolumeSpecName: "kube-api-access-b7bg2") pod "c1deb148-0a7b-454d-b2bd-fec2426dbbca" (UID: "c1deb148-0a7b-454d-b2bd-fec2426dbbca"). InnerVolumeSpecName "kube-api-access-b7bg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:46:33.612297 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:33.612258 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:46:33.612297 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:33.612286 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1deb148-0a7b-454d-b2bd-fec2426dbbca-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:46:33.612297 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:33.612296 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7bg2\" (UniqueName: \"kubernetes.io/projected/c1deb148-0a7b-454d-b2bd-fec2426dbbca-kube-api-access-b7bg2\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:46:33.612297 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:33.612305 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7t5b\" (UniqueName: \"kubernetes.io/projected/11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9-kube-api-access-k7t5b\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:46:34.409530 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:34.409428 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dfb9af4b-694f749f5d-qnmpf" Sep 09 10:46:34.409530 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:34.409461 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dfb9af4b-694f749f5d-qnmpf" event={"ID":"11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9","Type":"ContainerDied","Data":"45cf0baa167653ede1134f8fd42a47c4835ab903b31a299bde0e7061269ed7a9"} Sep 09 10:46:34.410027 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:34.409537 2559 scope.go:117] "RemoveContainer" containerID="7141409489e10c55cba2b5e0827189c20e111298ed263d4a5b2c9962d89a5db3" Sep 09 10:46:34.410969 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:34.410952 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dfb9af4b-6465749495-nx28m" Sep 09 10:46:34.411073 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:34.410970 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dfb9af4b-6465749495-nx28m" event={"ID":"c1deb148-0a7b-454d-b2bd-fec2426dbbca","Type":"ContainerDied","Data":"4adc8b38d7459d724e4a91dc5ef4aa97a41e94f22f6c097b6bc2696b84107dac"} Sep 09 10:46:34.418851 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:34.418830 2559 scope.go:117] "RemoveContainer" containerID="fb3c0efa8a07df37b5948971b5948e4db219ac16adc308be995fe2943fa881c7" Sep 09 10:46:34.428003 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:34.427981 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dfb9af4b-694f749f5d-qnmpf"] Sep 09 10:46:34.432294 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:34.432269 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dfb9af4b-694f749f5d-qnmpf"] Sep 09 10:46:34.446077 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:34.446042 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dfb9af4b-6465749495-nx28m"] Sep 09 10:46:34.450195 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:34.450165 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dfb9af4b-6465749495-nx28m"] Sep 09 10:46:35.070660 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:35.070624 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4bbfe3-c47867b4b-z6qd8"] Sep 09 10:46:35.070900 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:35.070870 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-4bbfe3-c47867b4b-z6qd8" podUID="659bf760-b7e1-4a04-96b5-d33c3acacb59" containerName="payload-pre-processing" containerID="cri-o://b5b57c67d6c1964fb21a1f9ee21b0c7054e63b2a35e06968f9bb43811f59cbc5" gracePeriod=30 Sep 09 10:46:35.333117 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:35.333085 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4bbfe3-fstc7"] Sep 09 10:46:35.336545 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:35.336521 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-4bbfe3-c47867b4b-z6qd8" Sep 09 10:46:35.346852 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:35.346815 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4bbfe3-fstc7"] Sep 09 10:46:35.424337 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:35.424300 2559 generic.go:356] "Generic (PLEG): container finished" podID="659bf760-b7e1-4a04-96b5-d33c3acacb59" containerID="b5b57c67d6c1964fb21a1f9ee21b0c7054e63b2a35e06968f9bb43811f59cbc5" exitCode=0 Sep 09 10:46:35.425397 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:35.424367 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-4bbfe3-c47867b4b-z6qd8" event={"ID":"659bf760-b7e1-4a04-96b5-d33c3acacb59","Type":"ContainerDied","Data":"b5b57c67d6c1964fb21a1f9ee21b0c7054e63b2a35e06968f9bb43811f59cbc5"} Sep 09 10:46:35.425397 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:35.424412 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-4bbfe3-c47867b4b-z6qd8" event={"ID":"659bf760-b7e1-4a04-96b5-d33c3acacb59","Type":"ContainerDied","Data":"5135f5fe5ad8baf8a50371a656366958dbc8b6da522ba66de0fd46e61ae76dca"} Sep 09 10:46:35.425397 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:35.424434 2559 scope.go:117] "RemoveContainer" containerID="b5b57c67d6c1964fb21a1f9ee21b0c7054e63b2a35e06968f9bb43811f59cbc5" Sep 09 10:46:35.425397 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:35.424370 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-4bbfe3-c47867b4b-z6qd8" Sep 09 10:46:35.428901 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:35.428878 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9vm9\" (UniqueName: \"kubernetes.io/projected/659bf760-b7e1-4a04-96b5-d33c3acacb59-kube-api-access-l9vm9\") pod \"659bf760-b7e1-4a04-96b5-d33c3acacb59\" (UID: \"659bf760-b7e1-4a04-96b5-d33c3acacb59\") " Sep 09 10:46:35.429312 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:35.428920 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/659bf760-b7e1-4a04-96b5-d33c3acacb59-plugins-config-volume\") pod \"659bf760-b7e1-4a04-96b5-d33c3acacb59\" (UID: \"659bf760-b7e1-4a04-96b5-d33c3acacb59\") " Sep 09 10:46:35.437839 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:35.437793 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/659bf760-b7e1-4a04-96b5-d33c3acacb59-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "659bf760-b7e1-4a04-96b5-d33c3acacb59" (UID: "659bf760-b7e1-4a04-96b5-d33c3acacb59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:46:35.439914 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:35.439886 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/659bf760-b7e1-4a04-96b5-d33c3acacb59-kube-api-access-l9vm9" (OuterVolumeSpecName: "kube-api-access-l9vm9") pod "659bf760-b7e1-4a04-96b5-d33c3acacb59" (UID: "659bf760-b7e1-4a04-96b5-d33c3acacb59"). InnerVolumeSpecName "kube-api-access-l9vm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:46:35.446415 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:35.446389 2559 scope.go:117] "RemoveContainer" containerID="b5b57c67d6c1964fb21a1f9ee21b0c7054e63b2a35e06968f9bb43811f59cbc5" Sep 09 10:46:35.446855 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:46:35.446832 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5b57c67d6c1964fb21a1f9ee21b0c7054e63b2a35e06968f9bb43811f59cbc5\": container with ID starting with b5b57c67d6c1964fb21a1f9ee21b0c7054e63b2a35e06968f9bb43811f59cbc5 not found: ID does not exist" containerID="b5b57c67d6c1964fb21a1f9ee21b0c7054e63b2a35e06968f9bb43811f59cbc5" Sep 09 10:46:35.446953 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:35.446864 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5b57c67d6c1964fb21a1f9ee21b0c7054e63b2a35e06968f9bb43811f59cbc5"} err="failed to get container status \"b5b57c67d6c1964fb21a1f9ee21b0c7054e63b2a35e06968f9bb43811f59cbc5\": rpc error: code = NotFound desc = could not find container \"b5b57c67d6c1964fb21a1f9ee21b0c7054e63b2a35e06968f9bb43811f59cbc5\": container with ID starting with b5b57c67d6c1964fb21a1f9ee21b0c7054e63b2a35e06968f9bb43811f59cbc5 not found: ID does not exist" Sep 09 10:46:35.532155 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:35.532119 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9vm9\" (UniqueName: \"kubernetes.io/projected/659bf760-b7e1-4a04-96b5-d33c3acacb59-kube-api-access-l9vm9\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:46:35.532155 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:35.532148 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/659bf760-b7e1-4a04-96b5-d33c3acacb59-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:46:35.749439 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:35.749398 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4bbfe3-c47867b4b-z6qd8"] Sep 09 10:46:35.756130 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:35.756100 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4bbfe3-c47867b4b-z6qd8"] Sep 09 10:46:36.334937 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:36.334899 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9" path="/var/lib/kubelet/pods/11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9/volumes" Sep 09 10:46:36.335270 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:36.335257 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="659bf760-b7e1-4a04-96b5-d33c3acacb59" path="/var/lib/kubelet/pods/659bf760-b7e1-4a04-96b5-d33c3acacb59/volumes" Sep 09 10:46:36.335608 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:36.335595 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1deb148-0a7b-454d-b2bd-fec2426dbbca" path="/var/lib/kubelet/pods/c1deb148-0a7b-454d-b2bd-fec2426dbbca/volumes" Sep 09 10:46:36.335880 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:36.335869 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0b60b4c-8233-4918-b268-e544239894c4" path="/var/lib/kubelet/pods/e0b60b4c-8233-4918-b268-e544239894c4/volumes" Sep 09 10:46:40.139906 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:40.139872 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d84c5166-5c4f84b8b9-qgjpv"] Sep 09 10:46:40.140281 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:40.140236 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="659bf760-b7e1-4a04-96b5-d33c3acacb59" containerName="payload-pre-processing" Sep 09 10:46:40.140281 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:40.140249 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="659bf760-b7e1-4a04-96b5-d33c3acacb59" containerName="payload-pre-processing" Sep 09 10:46:40.140281 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:40.140267 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0b60b4c-8233-4918-b268-e544239894c4" containerName="revoke-keys" Sep 09 10:46:40.140281 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:40.140272 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0b60b4c-8233-4918-b268-e544239894c4" containerName="revoke-keys" Sep 09 10:46:40.140405 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:40.140284 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1deb148-0a7b-454d-b2bd-fec2426dbbca" containerName="payload-processing" Sep 09 10:46:40.140405 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:40.140290 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1deb148-0a7b-454d-b2bd-fec2426dbbca" containerName="payload-processing" Sep 09 10:46:40.140405 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:40.140299 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9" containerName="payload-pre-processing" Sep 09 10:46:40.140405 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:40.140304 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9" containerName="payload-pre-processing" Sep 09 10:46:40.140405 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:40.140374 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1deb148-0a7b-454d-b2bd-fec2426dbbca" containerName="payload-processing" Sep 09 10:46:40.140405 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:40.140382 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0b60b4c-8233-4918-b268-e544239894c4" containerName="revoke-keys" Sep 09 10:46:40.140405 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:40.140391 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="11fa8a5c-ea9d-49c4-9a7c-25cbbc86d8a9" containerName="payload-pre-processing" Sep 09 10:46:40.140405 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:40.140399 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="659bf760-b7e1-4a04-96b5-d33c3acacb59" containerName="payload-pre-processing" Sep 09 10:46:40.144892 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:40.144868 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d84c5166-5c4f84b8b9-qgjpv" Sep 09 10:46:40.148627 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:40.148606 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-d84c5166-dockercfg-jj29k\"" Sep 09 10:46:40.149200 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:40.149182 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-d84c5166\"" Sep 09 10:46:40.163126 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:40.163095 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d84c5166-5c4f84b8b9-qgjpv"] Sep 09 10:46:40.274001 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:40.273959 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/562f0745-0d2d-4c9e-8b82-1df2c7cf6b10-plugins-config-volume\") pod \"payload-processing-e2e-derive-d84c5166-5c4f84b8b9-qgjpv\" (UID: \"562f0745-0d2d-4c9e-8b82-1df2c7cf6b10\") " pod="openshift-ingress/payload-processing-e2e-derive-d84c5166-5c4f84b8b9-qgjpv" Sep 09 10:46:40.274001 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:40.274001 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h47mh\" (UniqueName: \"kubernetes.io/projected/562f0745-0d2d-4c9e-8b82-1df2c7cf6b10-kube-api-access-h47mh\") pod \"payload-processing-e2e-derive-d84c5166-5c4f84b8b9-qgjpv\" (UID: \"562f0745-0d2d-4c9e-8b82-1df2c7cf6b10\") " pod="openshift-ingress/payload-processing-e2e-derive-d84c5166-5c4f84b8b9-qgjpv" Sep 09 10:46:40.375208 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:40.375168 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/562f0745-0d2d-4c9e-8b82-1df2c7cf6b10-plugins-config-volume\") pod \"payload-processing-e2e-derive-d84c5166-5c4f84b8b9-qgjpv\" (UID: \"562f0745-0d2d-4c9e-8b82-1df2c7cf6b10\") " pod="openshift-ingress/payload-processing-e2e-derive-d84c5166-5c4f84b8b9-qgjpv" Sep 09 10:46:40.375208 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:40.375209 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h47mh\" (UniqueName: \"kubernetes.io/projected/562f0745-0d2d-4c9e-8b82-1df2c7cf6b10-kube-api-access-h47mh\") pod \"payload-processing-e2e-derive-d84c5166-5c4f84b8b9-qgjpv\" (UID: \"562f0745-0d2d-4c9e-8b82-1df2c7cf6b10\") " pod="openshift-ingress/payload-processing-e2e-derive-d84c5166-5c4f84b8b9-qgjpv" Sep 09 10:46:40.375903 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:40.375881 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/562f0745-0d2d-4c9e-8b82-1df2c7cf6b10-plugins-config-volume\") pod \"payload-processing-e2e-derive-d84c5166-5c4f84b8b9-qgjpv\" (UID: \"562f0745-0d2d-4c9e-8b82-1df2c7cf6b10\") " pod="openshift-ingress/payload-processing-e2e-derive-d84c5166-5c4f84b8b9-qgjpv" Sep 09 10:46:40.385453 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:40.385426 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h47mh\" (UniqueName: \"kubernetes.io/projected/562f0745-0d2d-4c9e-8b82-1df2c7cf6b10-kube-api-access-h47mh\") pod \"payload-processing-e2e-derive-d84c5166-5c4f84b8b9-qgjpv\" (UID: \"562f0745-0d2d-4c9e-8b82-1df2c7cf6b10\") " pod="openshift-ingress/payload-processing-e2e-derive-d84c5166-5c4f84b8b9-qgjpv" Sep 09 10:46:40.455838 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:40.455758 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d84c5166-5c4f84b8b9-qgjpv" Sep 09 10:46:40.730851 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:40.730820 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d84c5166-5c4f84b8b9-qgjpv"] Sep 09 10:46:40.733659 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:46:40.733633 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod562f0745_0d2d_4c9e_8b82_1df2c7cf6b10.slice/crio-709dbc80372eed4aadc92df8b0f592d61c249d449a387c91def3296807078cc3 WatchSource:0}: Error finding container 709dbc80372eed4aadc92df8b0f592d61c249d449a387c91def3296807078cc3: Status 404 returned error can't find the container with id 709dbc80372eed4aadc92df8b0f592d61c249d449a387c91def3296807078cc3 Sep 09 10:46:41.460444 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:41.460405 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d84c5166-5c4f84b8b9-qgjpv" event={"ID":"562f0745-0d2d-4c9e-8b82-1df2c7cf6b10","Type":"ContainerStarted","Data":"8eaf6211fcd3859d802eab79eadf362a70f83dfe8020242f4b59a61b2efbac7e"} Sep 09 10:46:41.460852 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:41.460467 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d84c5166-5c4f84b8b9-qgjpv" event={"ID":"562f0745-0d2d-4c9e-8b82-1df2c7cf6b10","Type":"ContainerStarted","Data":"709dbc80372eed4aadc92df8b0f592d61c249d449a387c91def3296807078cc3"} Sep 09 10:46:41.460852 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:41.460493 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-d84c5166-5c4f84b8b9-qgjpv" Sep 09 10:46:41.481056 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:41.481001 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-d84c5166-5c4f84b8b9-qgjpv" podStartSLOduration=1.4809824200000001 podStartE2EDuration="1.48098242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:46:41.479829323 +0000 UTC m=+1241.654431648" watchObservedRunningTime="2026-09-09 10:46:41.48098242 +0000 UTC m=+1241.655584746" Sep 09 10:46:47.092866 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:47.092814 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" podUID="9b179542-2be1-4dad-9097-f3b4bc0d2dcb" containerName="main" containerID="cri-o://dc642b7be06a53ecd50ac4d45f1118f6e9421d7c19fab01040d09e870c5df1fc" gracePeriod=45 Sep 09 10:46:47.346517 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:47.346424 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" Sep 09 10:46:47.439781 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:47.439742 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-tls-certs\") pod \"9b179542-2be1-4dad-9097-f3b4bc0d2dcb\" (UID: \"9b179542-2be1-4dad-9097-f3b4bc0d2dcb\") " Sep 09 10:46:47.439962 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:47.439850 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7drz\" (UniqueName: \"kubernetes.io/projected/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-kube-api-access-f7drz\") pod \"9b179542-2be1-4dad-9097-f3b4bc0d2dcb\" (UID: \"9b179542-2be1-4dad-9097-f3b4bc0d2dcb\") " Sep 09 10:46:47.439962 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:47.439875 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-dshm\") pod \"9b179542-2be1-4dad-9097-f3b4bc0d2dcb\" (UID: \"9b179542-2be1-4dad-9097-f3b4bc0d2dcb\") " Sep 09 10:46:47.439962 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:47.439912 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-tmp-dir\") pod \"9b179542-2be1-4dad-9097-f3b4bc0d2dcb\" (UID: \"9b179542-2be1-4dad-9097-f3b4bc0d2dcb\") " Sep 09 10:46:47.440159 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:47.440058 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-model-cache\") pod \"9b179542-2be1-4dad-9097-f3b4bc0d2dcb\" (UID: \"9b179542-2be1-4dad-9097-f3b4bc0d2dcb\") " Sep 09 10:46:47.440159 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:47.440105 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-home\") pod \"9b179542-2be1-4dad-9097-f3b4bc0d2dcb\" (UID: \"9b179542-2be1-4dad-9097-f3b4bc0d2dcb\") " Sep 09 10:46:47.440275 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:47.440188 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b179542-2be1-4dad-9097-f3b4bc0d2dcb" (UID: "9b179542-2be1-4dad-9097-f3b4bc0d2dcb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 10:46:47.440365 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:47.440341 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-home" (OuterVolumeSpecName: "home") pod "9b179542-2be1-4dad-9097-f3b4bc0d2dcb" (UID: "9b179542-2be1-4dad-9097-f3b4bc0d2dcb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 10:46:47.440365 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:47.440352 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b179542-2be1-4dad-9097-f3b4bc0d2dcb" (UID: "9b179542-2be1-4dad-9097-f3b4bc0d2dcb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 10:46:47.440475 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:47.440428 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-tmp-dir\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:46:47.440475 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:47.440446 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-model-cache\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:46:47.440475 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:47.440454 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-home\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:46:47.442837 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:47.442802 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-dshm" (OuterVolumeSpecName: "dshm") pod "9b179542-2be1-4dad-9097-f3b4bc0d2dcb" (UID: "9b179542-2be1-4dad-9097-f3b4bc0d2dcb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 10:46:47.442995 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:47.442976 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b179542-2be1-4dad-9097-f3b4bc0d2dcb" (UID: "9b179542-2be1-4dad-9097-f3b4bc0d2dcb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:46:47.443067 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:47.443048 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-kube-api-access-f7drz" (OuterVolumeSpecName: "kube-api-access-f7drz") pod "9b179542-2be1-4dad-9097-f3b4bc0d2dcb" (UID: "9b179542-2be1-4dad-9097-f3b4bc0d2dcb"). InnerVolumeSpecName "kube-api-access-f7drz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:46:47.492282 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:47.492243 2559 generic.go:356] "Generic (PLEG): container finished" podID="9b179542-2be1-4dad-9097-f3b4bc0d2dcb" containerID="dc642b7be06a53ecd50ac4d45f1118f6e9421d7c19fab01040d09e870c5df1fc" exitCode=0 Sep 09 10:46:47.492479 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:47.492292 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" Sep 09 10:46:47.492479 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:47.492296 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" event={"ID":"9b179542-2be1-4dad-9097-f3b4bc0d2dcb","Type":"ContainerDied","Data":"dc642b7be06a53ecd50ac4d45f1118f6e9421d7c19fab01040d09e870c5df1fc"} Sep 09 10:46:47.492479 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:47.492412 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6" event={"ID":"9b179542-2be1-4dad-9097-f3b4bc0d2dcb","Type":"ContainerDied","Data":"e0053a02c77802bb2f1bfda2c6f414ad4039f87a716a65f890b1fc9da376c025"} Sep 09 10:46:47.492479 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:47.492436 2559 scope.go:117] "RemoveContainer" containerID="dc642b7be06a53ecd50ac4d45f1118f6e9421d7c19fab01040d09e870c5df1fc" Sep 09 10:46:47.505120 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:47.505085 2559 scope.go:117] "RemoveContainer" containerID="dc642b7be06a53ecd50ac4d45f1118f6e9421d7c19fab01040d09e870c5df1fc" Sep 09 10:46:47.505429 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:46:47.505408 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc642b7be06a53ecd50ac4d45f1118f6e9421d7c19fab01040d09e870c5df1fc\": container with ID starting with dc642b7be06a53ecd50ac4d45f1118f6e9421d7c19fab01040d09e870c5df1fc not found: ID does not exist" containerID="dc642b7be06a53ecd50ac4d45f1118f6e9421d7c19fab01040d09e870c5df1fc" Sep 09 10:46:47.505526 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:47.505442 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc642b7be06a53ecd50ac4d45f1118f6e9421d7c19fab01040d09e870c5df1fc"} err="failed to get container status \"dc642b7be06a53ecd50ac4d45f1118f6e9421d7c19fab01040d09e870c5df1fc\": rpc error: code = NotFound desc = could not find container \"dc642b7be06a53ecd50ac4d45f1118f6e9421d7c19fab01040d09e870c5df1fc\": container with ID starting with dc642b7be06a53ecd50ac4d45f1118f6e9421d7c19fab01040d09e870c5df1fc not found: ID does not exist" Sep 09 10:46:47.522883 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:47.522840 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6"] Sep 09 10:46:47.528287 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:47.528253 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-4bbfe3/autores-model-4bbfe3-kserve-7c6876776c-r4pm6"] Sep 09 10:46:47.541663 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:47.541629 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-tls-certs\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:46:47.541663 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:47.541660 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7drz\" (UniqueName: \"kubernetes.io/projected/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-kube-api-access-f7drz\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:46:47.541663 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:47.541671 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b179542-2be1-4dad-9097-f3b4bc0d2dcb-dshm\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:46:48.334109 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:48.334071 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b179542-2be1-4dad-9097-f3b4bc0d2dcb" path="/var/lib/kubelet/pods/9b179542-2be1-4dad-9097-f3b4bc0d2dcb/volumes" Sep 09 10:46:52.468136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:46:52.468063 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-d84c5166-5c4f84b8b9-qgjpv" Sep 09 10:47:03.115899 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:03.115859 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d84c5166-g4jdq"] Sep 09 10:47:03.116297 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:03.116214 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b179542-2be1-4dad-9097-f3b4bc0d2dcb" containerName="main" Sep 09 10:47:03.116297 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:03.116225 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b179542-2be1-4dad-9097-f3b4bc0d2dcb" containerName="main" Sep 09 10:47:03.116297 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:03.116294 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b179542-2be1-4dad-9097-f3b4bc0d2dcb" containerName="main" Sep 09 10:47:03.120808 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:03.120787 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d84c5166-g4jdq" Sep 09 10:47:03.124362 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:03.124338 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 10:47:03.124482 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:03.124384 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kq7qv\"" Sep 09 10:47:03.131367 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:03.131337 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d84c5166-g4jdq"] Sep 09 10:47:03.279179 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:03.279142 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a3f2083-d1b1-4467-9176-f9188e510895-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d84c5166-g4jdq\" (UID: \"9a3f2083-d1b1-4467-9176-f9188e510895\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d84c5166-g4jdq" Sep 09 10:47:03.379901 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:03.379810 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a3f2083-d1b1-4467-9176-f9188e510895-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d84c5166-g4jdq\" (UID: \"9a3f2083-d1b1-4467-9176-f9188e510895\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d84c5166-g4jdq" Sep 09 10:47:03.380434 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:03.380412 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a3f2083-d1b1-4467-9176-f9188e510895-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d84c5166-g4jdq\" (UID: \"9a3f2083-d1b1-4467-9176-f9188e510895\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d84c5166-g4jdq" Sep 09 10:47:03.433097 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:03.433059 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d84c5166-g4jdq" Sep 09 10:47:03.710089 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:03.710064 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d84c5166-g4jdq"] Sep 09 10:47:03.713298 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:47:03.713267 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a3f2083_d1b1_4467_9176_f9188e510895.slice/crio-4e618a11ec7b3433e3e21bc7e9982a95ad640673bd969ff5e37b83ce1b9b5ce7 WatchSource:0}: Error finding container 4e618a11ec7b3433e3e21bc7e9982a95ad640673bd969ff5e37b83ce1b9b5ce7: Status 404 returned error can't find the container with id 4e618a11ec7b3433e3e21bc7e9982a95ad640673bd969ff5e37b83ce1b9b5ce7 Sep 09 10:47:04.583779 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:04.583740 2559 generic.go:356] "Generic (PLEG): container finished" podID="9a3f2083-d1b1-4467-9176-f9188e510895" containerID="83936076cd166efe6fbc77f4c5b4147a8655f5078a04470eb3842723a3f98300" exitCode=0 Sep 09 10:47:04.584206 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:04.583831 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d84c5166-g4jdq" event={"ID":"9a3f2083-d1b1-4467-9176-f9188e510895","Type":"ContainerDied","Data":"83936076cd166efe6fbc77f4c5b4147a8655f5078a04470eb3842723a3f98300"} Sep 09 10:47:04.584206 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:04.583881 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d84c5166-g4jdq" event={"ID":"9a3f2083-d1b1-4467-9176-f9188e510895","Type":"ContainerStarted","Data":"4e618a11ec7b3433e3e21bc7e9982a95ad640673bd969ff5e37b83ce1b9b5ce7"} Sep 09 10:47:05.722042 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:05.722017 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d84c5166-g4jdq" Sep 09 10:47:05.800071 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:05.800023 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a3f2083-d1b1-4467-9176-f9188e510895-maas-api-service-ca\") pod \"9a3f2083-d1b1-4467-9176-f9188e510895\" (UID: \"9a3f2083-d1b1-4467-9176-f9188e510895\") " Sep 09 10:47:05.800424 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:05.800397 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a3f2083-d1b1-4467-9176-f9188e510895-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9a3f2083-d1b1-4467-9176-f9188e510895" (UID: "9a3f2083-d1b1-4467-9176-f9188e510895"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:47:05.901661 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:05.901575 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a3f2083-d1b1-4467-9176-f9188e510895-maas-api-service-ca\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:47:06.595755 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:06.595663 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d84c5166-g4jdq" Sep 09 10:47:06.595755 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:06.595705 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d84c5166-g4jdq" event={"ID":"9a3f2083-d1b1-4467-9176-f9188e510895","Type":"ContainerDied","Data":"4e618a11ec7b3433e3e21bc7e9982a95ad640673bd969ff5e37b83ce1b9b5ce7"} Sep 09 10:47:06.595946 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:06.595761 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e618a11ec7b3433e3e21bc7e9982a95ad640673bd969ff5e37b83ce1b9b5ce7" Sep 09 10:47:10.122875 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:10.122833 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d84c5166-5c4f84b8b9-qgjpv"] Sep 09 10:47:10.123353 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:10.123109 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-d84c5166-5c4f84b8b9-qgjpv" podUID="562f0745-0d2d-4c9e-8b82-1df2c7cf6b10" containerName="payload-processing" containerID="cri-o://8eaf6211fcd3859d802eab79eadf362a70f83dfe8020242f4b59a61b2efbac7e" gracePeriod=30 Sep 09 10:47:10.381063 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:10.380996 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d84c5166-5c4f84b8b9-qgjpv" Sep 09 10:47:10.524751 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:10.524720 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d84c5166-g4jdq"] Sep 09 10:47:10.527433 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:10.527406 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d84c5166-g4jdq"] Sep 09 10:47:10.545912 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:10.545872 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h47mh\" (UniqueName: \"kubernetes.io/projected/562f0745-0d2d-4c9e-8b82-1df2c7cf6b10-kube-api-access-h47mh\") pod \"562f0745-0d2d-4c9e-8b82-1df2c7cf6b10\" (UID: \"562f0745-0d2d-4c9e-8b82-1df2c7cf6b10\") " Sep 09 10:47:10.546088 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:10.545975 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/562f0745-0d2d-4c9e-8b82-1df2c7cf6b10-plugins-config-volume\") pod \"562f0745-0d2d-4c9e-8b82-1df2c7cf6b10\" (UID: \"562f0745-0d2d-4c9e-8b82-1df2c7cf6b10\") " Sep 09 10:47:10.546573 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:10.546538 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/562f0745-0d2d-4c9e-8b82-1df2c7cf6b10-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "562f0745-0d2d-4c9e-8b82-1df2c7cf6b10" (UID: "562f0745-0d2d-4c9e-8b82-1df2c7cf6b10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:47:10.548217 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:10.548186 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/562f0745-0d2d-4c9e-8b82-1df2c7cf6b10-kube-api-access-h47mh" (OuterVolumeSpecName: "kube-api-access-h47mh") pod "562f0745-0d2d-4c9e-8b82-1df2c7cf6b10" (UID: "562f0745-0d2d-4c9e-8b82-1df2c7cf6b10"). InnerVolumeSpecName "kube-api-access-h47mh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:47:10.617762 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:10.617723 2559 generic.go:356] "Generic (PLEG): container finished" podID="562f0745-0d2d-4c9e-8b82-1df2c7cf6b10" containerID="8eaf6211fcd3859d802eab79eadf362a70f83dfe8020242f4b59a61b2efbac7e" exitCode=0 Sep 09 10:47:10.617945 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:10.617770 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d84c5166-5c4f84b8b9-qgjpv" Sep 09 10:47:10.617945 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:10.617816 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d84c5166-5c4f84b8b9-qgjpv" event={"ID":"562f0745-0d2d-4c9e-8b82-1df2c7cf6b10","Type":"ContainerDied","Data":"8eaf6211fcd3859d802eab79eadf362a70f83dfe8020242f4b59a61b2efbac7e"} Sep 09 10:47:10.617945 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:10.617875 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d84c5166-5c4f84b8b9-qgjpv" event={"ID":"562f0745-0d2d-4c9e-8b82-1df2c7cf6b10","Type":"ContainerDied","Data":"709dbc80372eed4aadc92df8b0f592d61c249d449a387c91def3296807078cc3"} Sep 09 10:47:10.617945 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:10.617903 2559 scope.go:117] "RemoveContainer" containerID="8eaf6211fcd3859d802eab79eadf362a70f83dfe8020242f4b59a61b2efbac7e" Sep 09 10:47:10.635357 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:10.635338 2559 scope.go:117] "RemoveContainer" containerID="8eaf6211fcd3859d802eab79eadf362a70f83dfe8020242f4b59a61b2efbac7e" Sep 09 10:47:10.635707 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:47:10.635686 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eaf6211fcd3859d802eab79eadf362a70f83dfe8020242f4b59a61b2efbac7e\": container with ID starting with 8eaf6211fcd3859d802eab79eadf362a70f83dfe8020242f4b59a61b2efbac7e not found: ID does not exist" containerID="8eaf6211fcd3859d802eab79eadf362a70f83dfe8020242f4b59a61b2efbac7e" Sep 09 10:47:10.635786 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:10.635715 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eaf6211fcd3859d802eab79eadf362a70f83dfe8020242f4b59a61b2efbac7e"} err="failed to get container status \"8eaf6211fcd3859d802eab79eadf362a70f83dfe8020242f4b59a61b2efbac7e\": rpc error: code = NotFound desc = could not find container \"8eaf6211fcd3859d802eab79eadf362a70f83dfe8020242f4b59a61b2efbac7e\": container with ID starting with 8eaf6211fcd3859d802eab79eadf362a70f83dfe8020242f4b59a61b2efbac7e not found: ID does not exist" Sep 09 10:47:10.647192 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:10.647160 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d84c5166-5c4f84b8b9-qgjpv"] Sep 09 10:47:10.651337 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:10.651310 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d84c5166-5c4f84b8b9-qgjpv"] Sep 09 10:47:10.651550 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:10.647975 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/562f0745-0d2d-4c9e-8b82-1df2c7cf6b10-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:47:10.651688 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:10.651559 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h47mh\" (UniqueName: \"kubernetes.io/projected/562f0745-0d2d-4c9e-8b82-1df2c7cf6b10-kube-api-access-h47mh\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:47:12.337917 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:12.337874 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="562f0745-0d2d-4c9e-8b82-1df2c7cf6b10" path="/var/lib/kubelet/pods/562f0745-0d2d-4c9e-8b82-1df2c7cf6b10/volumes" Sep 09 10:47:12.338403 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:12.338364 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a3f2083-d1b1-4467-9176-f9188e510895" path="/var/lib/kubelet/pods/9a3f2083-d1b1-4467-9176-f9188e510895/volumes" Sep 09 10:47:24.344177 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:24.344139 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-928d2a-57cd5d8fc7-wclgv"] Sep 09 10:47:24.344661 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:24.344542 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a3f2083-d1b1-4467-9176-f9188e510895" containerName="revoke-keys" Sep 09 10:47:24.344661 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:24.344556 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a3f2083-d1b1-4467-9176-f9188e510895" containerName="revoke-keys" Sep 09 10:47:24.344661 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:24.344583 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="562f0745-0d2d-4c9e-8b82-1df2c7cf6b10" containerName="payload-processing" Sep 09 10:47:24.344661 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:24.344590 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="562f0745-0d2d-4c9e-8b82-1df2c7cf6b10" containerName="payload-processing" Sep 09 10:47:24.344888 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:24.344671 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a3f2083-d1b1-4467-9176-f9188e510895" containerName="revoke-keys" Sep 09 10:47:24.344888 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:24.344682 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="562f0745-0d2d-4c9e-8b82-1df2c7cf6b10" containerName="payload-processing" Sep 09 10:47:24.349292 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:24.349267 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-928d2a-57cd5d8fc7-wclgv" Sep 09 10:47:24.352045 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:24.352017 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-928d2a\"" Sep 09 10:47:24.364032 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:24.364002 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-928d2a-57cd5d8fc7-wclgv"] Sep 09 10:47:24.377343 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:24.371946 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89f95a27-8b8f-4902-8092-f7dd8744cd82-plugins-config-volume\") pod \"payload-processing-e2e-explicit-928d2a-57cd5d8fc7-wclgv\" (UID: \"89f95a27-8b8f-4902-8092-f7dd8744cd82\") " pod="openshift-ingress/payload-processing-e2e-explicit-928d2a-57cd5d8fc7-wclgv" Sep 09 10:47:24.377343 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:24.372028 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjl6l\" (UniqueName: \"kubernetes.io/projected/89f95a27-8b8f-4902-8092-f7dd8744cd82-kube-api-access-mjl6l\") pod \"payload-processing-e2e-explicit-928d2a-57cd5d8fc7-wclgv\" (UID: \"89f95a27-8b8f-4902-8092-f7dd8744cd82\") " pod="openshift-ingress/payload-processing-e2e-explicit-928d2a-57cd5d8fc7-wclgv" Sep 09 10:47:24.399889 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:24.399846 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-928d2a-68487959c8-t86mr"] Sep 09 10:47:24.403441 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:24.403416 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-928d2a-68487959c8-t86mr" Sep 09 10:47:24.407289 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:24.407263 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-928d2a-dockercfg-p4hw9\"" Sep 09 10:47:24.424170 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:24.424136 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-928d2a-68487959c8-t86mr"] Sep 09 10:47:24.473257 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:24.473224 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb3c8fef-1358-4580-8bbb-c87af9d7500e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-928d2a-68487959c8-t86mr\" (UID: \"bb3c8fef-1358-4580-8bbb-c87af9d7500e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-928d2a-68487959c8-t86mr" Sep 09 10:47:24.473257 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:24.473260 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z6xh\" (UniqueName: \"kubernetes.io/projected/bb3c8fef-1358-4580-8bbb-c87af9d7500e-kube-api-access-8z6xh\") pod \"payload-pre-processing-e2e-explicit-928d2a-68487959c8-t86mr\" (UID: \"bb3c8fef-1358-4580-8bbb-c87af9d7500e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-928d2a-68487959c8-t86mr" Sep 09 10:47:24.473528 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:24.473295 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89f95a27-8b8f-4902-8092-f7dd8744cd82-plugins-config-volume\") pod \"payload-processing-e2e-explicit-928d2a-57cd5d8fc7-wclgv\" (UID: \"89f95a27-8b8f-4902-8092-f7dd8744cd82\") " pod="openshift-ingress/payload-processing-e2e-explicit-928d2a-57cd5d8fc7-wclgv" Sep 09 10:47:24.473528 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:24.473380 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjl6l\" (UniqueName: \"kubernetes.io/projected/89f95a27-8b8f-4902-8092-f7dd8744cd82-kube-api-access-mjl6l\") pod \"payload-processing-e2e-explicit-928d2a-57cd5d8fc7-wclgv\" (UID: \"89f95a27-8b8f-4902-8092-f7dd8744cd82\") " pod="openshift-ingress/payload-processing-e2e-explicit-928d2a-57cd5d8fc7-wclgv" Sep 09 10:47:24.474058 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:24.474035 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89f95a27-8b8f-4902-8092-f7dd8744cd82-plugins-config-volume\") pod \"payload-processing-e2e-explicit-928d2a-57cd5d8fc7-wclgv\" (UID: \"89f95a27-8b8f-4902-8092-f7dd8744cd82\") " pod="openshift-ingress/payload-processing-e2e-explicit-928d2a-57cd5d8fc7-wclgv" Sep 09 10:47:24.483635 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:24.483610 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjl6l\" (UniqueName: \"kubernetes.io/projected/89f95a27-8b8f-4902-8092-f7dd8744cd82-kube-api-access-mjl6l\") pod \"payload-processing-e2e-explicit-928d2a-57cd5d8fc7-wclgv\" (UID: \"89f95a27-8b8f-4902-8092-f7dd8744cd82\") " pod="openshift-ingress/payload-processing-e2e-explicit-928d2a-57cd5d8fc7-wclgv" Sep 09 10:47:24.574819 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:24.574775 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb3c8fef-1358-4580-8bbb-c87af9d7500e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-928d2a-68487959c8-t86mr\" (UID: \"bb3c8fef-1358-4580-8bbb-c87af9d7500e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-928d2a-68487959c8-t86mr" Sep 09 10:47:24.574819 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:24.574816 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z6xh\" (UniqueName: \"kubernetes.io/projected/bb3c8fef-1358-4580-8bbb-c87af9d7500e-kube-api-access-8z6xh\") pod \"payload-pre-processing-e2e-explicit-928d2a-68487959c8-t86mr\" (UID: \"bb3c8fef-1358-4580-8bbb-c87af9d7500e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-928d2a-68487959c8-t86mr" Sep 09 10:47:24.575531 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:24.575476 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb3c8fef-1358-4580-8bbb-c87af9d7500e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-928d2a-68487959c8-t86mr\" (UID: \"bb3c8fef-1358-4580-8bbb-c87af9d7500e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-928d2a-68487959c8-t86mr" Sep 09 10:47:24.585069 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:24.585045 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z6xh\" (UniqueName: \"kubernetes.io/projected/bb3c8fef-1358-4580-8bbb-c87af9d7500e-kube-api-access-8z6xh\") pod \"payload-pre-processing-e2e-explicit-928d2a-68487959c8-t86mr\" (UID: \"bb3c8fef-1358-4580-8bbb-c87af9d7500e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-928d2a-68487959c8-t86mr" Sep 09 10:47:24.659385 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:24.659297 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-928d2a-57cd5d8fc7-wclgv" Sep 09 10:47:24.713768 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:24.713740 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-928d2a-68487959c8-t86mr" Sep 09 10:47:24.937908 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:24.937878 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-928d2a-57cd5d8fc7-wclgv"] Sep 09 10:47:24.939513 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:47:24.939478 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89f95a27_8b8f_4902_8092_f7dd8744cd82.slice/crio-7a07033acea1eea3ae520498827a9b8ea041b484477db291fb3e62d9f99f86b9 WatchSource:0}: Error finding container 7a07033acea1eea3ae520498827a9b8ea041b484477db291fb3e62d9f99f86b9: Status 404 returned error can't find the container with id 7a07033acea1eea3ae520498827a9b8ea041b484477db291fb3e62d9f99f86b9 Sep 09 10:47:25.001512 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:25.001470 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-928d2a-68487959c8-t86mr"] Sep 09 10:47:25.003911 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:47:25.003878 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3c8fef_1358_4580_8bbb_c87af9d7500e.slice/crio-0233d55c7dabff5276928092cd3fa078ad605880c868bd89daba680a326552b4 WatchSource:0}: Error finding container 0233d55c7dabff5276928092cd3fa078ad605880c868bd89daba680a326552b4: Status 404 returned error can't find the container with id 0233d55c7dabff5276928092cd3fa078ad605880c868bd89daba680a326552b4 Sep 09 10:47:25.702104 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:25.702065 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-928d2a-68487959c8-t86mr" event={"ID":"bb3c8fef-1358-4580-8bbb-c87af9d7500e","Type":"ContainerStarted","Data":"9d8a9d04f03e259e1c518a7ceb0cc47c7b583d176a056d2c821255c31756817d"} Sep 09 10:47:25.702579 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:25.702120 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-928d2a-68487959c8-t86mr" event={"ID":"bb3c8fef-1358-4580-8bbb-c87af9d7500e","Type":"ContainerStarted","Data":"0233d55c7dabff5276928092cd3fa078ad605880c868bd89daba680a326552b4"} Sep 09 10:47:25.702579 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:25.702231 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-928d2a-68487959c8-t86mr" Sep 09 10:47:25.703784 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:25.703757 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-928d2a-57cd5d8fc7-wclgv" event={"ID":"89f95a27-8b8f-4902-8092-f7dd8744cd82","Type":"ContainerStarted","Data":"35cd6f4a56482a1fe6bad98c914229333c01061626957af046eef5e974e5b156"} Sep 09 10:47:25.703902 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:25.703798 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-928d2a-57cd5d8fc7-wclgv" event={"ID":"89f95a27-8b8f-4902-8092-f7dd8744cd82","Type":"ContainerStarted","Data":"7a07033acea1eea3ae520498827a9b8ea041b484477db291fb3e62d9f99f86b9"} Sep 09 10:47:25.703902 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:25.703844 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-928d2a-57cd5d8fc7-wclgv" Sep 09 10:47:25.725666 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:25.725614 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-928d2a-68487959c8-t86mr" podStartSLOduration=1.725596275 podStartE2EDuration="1.725596275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:47:25.724561296 +0000 UTC m=+1285.899163622" watchObservedRunningTime="2026-09-09 10:47:25.725596275 +0000 UTC m=+1285.900198599" Sep 09 10:47:25.758002 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:25.757944 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-928d2a-57cd5d8fc7-wclgv" podStartSLOduration=1.757924549 podStartE2EDuration="1.757924549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:47:25.755954319 +0000 UTC m=+1285.930556639" watchObservedRunningTime="2026-09-09 10:47:25.757924549 +0000 UTC m=+1285.932526874" Sep 09 10:47:36.712009 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:36.711977 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-928d2a-68487959c8-t86mr" Sep 09 10:47:36.712527 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:47:36.712057 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-928d2a-57cd5d8fc7-wclgv" Sep 09 10:48:03.778778 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:03.778746 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-928d2a-9p2r2"] Sep 09 10:48:03.783608 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:03.783584 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-928d2a-9p2r2" Sep 09 10:48:03.786292 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:03.786258 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 10:48:03.786424 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:03.786331 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kq7qv\"" Sep 09 10:48:03.791058 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:03.791021 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-928d2a-9p2r2"] Sep 09 10:48:03.924003 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:03.923968 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/144fca0b-75fb-43fe-9c89-1be4e5283bf4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-928d2a-9p2r2\" (UID: \"144fca0b-75fb-43fe-9c89-1be4e5283bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-928d2a-9p2r2" Sep 09 10:48:04.024993 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:04.024943 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/144fca0b-75fb-43fe-9c89-1be4e5283bf4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-928d2a-9p2r2\" (UID: \"144fca0b-75fb-43fe-9c89-1be4e5283bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-928d2a-9p2r2" Sep 09 10:48:04.025625 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:04.025604 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/144fca0b-75fb-43fe-9c89-1be4e5283bf4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-928d2a-9p2r2\" (UID: \"144fca0b-75fb-43fe-9c89-1be4e5283bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-928d2a-9p2r2" Sep 09 10:48:04.094867 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:04.094834 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-928d2a-9p2r2" Sep 09 10:48:04.369401 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:04.369373 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-928d2a-9p2r2"] Sep 09 10:48:04.371660 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:48:04.371634 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144fca0b_75fb_43fe_9c89_1be4e5283bf4.slice/crio-8d67172943c674f38f772f0c83cbe65a206738f9f04e388c5ec13a82715e36aa WatchSource:0}: Error finding container 8d67172943c674f38f772f0c83cbe65a206738f9f04e388c5ec13a82715e36aa: Status 404 returned error can't find the container with id 8d67172943c674f38f772f0c83cbe65a206738f9f04e388c5ec13a82715e36aa Sep 09 10:48:04.911824 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:04.911735 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-928d2a-9p2r2" event={"ID":"144fca0b-75fb-43fe-9c89-1be4e5283bf4","Type":"ContainerStarted","Data":"1959afb334def63288a376ff4d0a78fd56334c36e73d3d14c135dd08f1799cbd"} Sep 09 10:48:04.911824 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:04.911794 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-928d2a-9p2r2" event={"ID":"144fca0b-75fb-43fe-9c89-1be4e5283bf4","Type":"ContainerStarted","Data":"8d67172943c674f38f772f0c83cbe65a206738f9f04e388c5ec13a82715e36aa"} Sep 09 10:48:04.930925 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:04.930636 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-928d2a-9p2r2" podStartSLOduration=1.930620821 podStartE2EDuration="1.930620821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:48:04.929527854 +0000 UTC m=+1325.104130173" watchObservedRunningTime="2026-09-09 10:48:04.930620821 +0000 UTC m=+1325.105223146" Sep 09 10:48:05.918189 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:05.918153 2559 generic.go:356] "Generic (PLEG): container finished" podID="144fca0b-75fb-43fe-9c89-1be4e5283bf4" containerID="1959afb334def63288a376ff4d0a78fd56334c36e73d3d14c135dd08f1799cbd" exitCode=0 Sep 09 10:48:05.918628 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:05.918243 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-928d2a-9p2r2" event={"ID":"144fca0b-75fb-43fe-9c89-1be4e5283bf4","Type":"ContainerDied","Data":"1959afb334def63288a376ff4d0a78fd56334c36e73d3d14c135dd08f1799cbd"} Sep 09 10:48:07.065903 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:07.065870 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-928d2a-9p2r2" Sep 09 10:48:07.156434 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:07.156398 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/144fca0b-75fb-43fe-9c89-1be4e5283bf4-maas-api-service-ca\") pod \"144fca0b-75fb-43fe-9c89-1be4e5283bf4\" (UID: \"144fca0b-75fb-43fe-9c89-1be4e5283bf4\") " Sep 09 10:48:07.156820 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:07.156793 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/144fca0b-75fb-43fe-9c89-1be4e5283bf4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "144fca0b-75fb-43fe-9c89-1be4e5283bf4" (UID: "144fca0b-75fb-43fe-9c89-1be4e5283bf4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:48:07.257309 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:07.257225 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/144fca0b-75fb-43fe-9c89-1be4e5283bf4-maas-api-service-ca\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:48:07.929607 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:07.929575 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-928d2a-9p2r2" Sep 09 10:48:07.929607 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:07.929586 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-928d2a-9p2r2" event={"ID":"144fca0b-75fb-43fe-9c89-1be4e5283bf4","Type":"ContainerDied","Data":"8d67172943c674f38f772f0c83cbe65a206738f9f04e388c5ec13a82715e36aa"} Sep 09 10:48:07.929918 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:07.929648 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d67172943c674f38f772f0c83cbe65a206738f9f04e388c5ec13a82715e36aa" Sep 09 10:48:11.469584 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:11.469546 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40f1e9-knhnp"] Sep 09 10:48:11.470024 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:11.470002 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="144fca0b-75fb-43fe-9c89-1be4e5283bf4" containerName="revoke-keys" Sep 09 10:48:11.470024 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:11.470024 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="144fca0b-75fb-43fe-9c89-1be4e5283bf4" containerName="revoke-keys" Sep 09 10:48:11.470153 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:11.470104 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="144fca0b-75fb-43fe-9c89-1be4e5283bf4" containerName="revoke-keys" Sep 09 10:48:11.473357 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:11.473329 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40f1e9-knhnp" Sep 09 10:48:11.475905 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:11.475882 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 10:48:11.476043 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:11.475933 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kq7qv\"" Sep 09 10:48:11.484961 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:11.484512 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40f1e9-knhnp"] Sep 09 10:48:11.493592 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:11.493549 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddeb674b-b9af-40e9-861b-0aa0a3f216c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-40f1e9-knhnp\" (UID: \"ddeb674b-b9af-40e9-861b-0aa0a3f216c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40f1e9-knhnp" Sep 09 10:48:11.595075 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:11.595033 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddeb674b-b9af-40e9-861b-0aa0a3f216c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-40f1e9-knhnp\" (UID: \"ddeb674b-b9af-40e9-861b-0aa0a3f216c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40f1e9-knhnp" Sep 09 10:48:11.595727 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:11.595703 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddeb674b-b9af-40e9-861b-0aa0a3f216c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-40f1e9-knhnp\" (UID: \"ddeb674b-b9af-40e9-861b-0aa0a3f216c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40f1e9-knhnp" Sep 09 10:48:11.791044 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:11.790960 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40f1e9-knhnp" Sep 09 10:48:12.063185 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:12.063160 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40f1e9-knhnp"] Sep 09 10:48:12.065572 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:48:12.065541 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddeb674b_b9af_40e9_861b_0aa0a3f216c4.slice/crio-30bdb450b21ef03d1c6daeb12ba9fa5340a4e4841b741e0499c8a337985fafaf WatchSource:0}: Error finding container 30bdb450b21ef03d1c6daeb12ba9fa5340a4e4841b741e0499c8a337985fafaf: Status 404 returned error can't find the container with id 30bdb450b21ef03d1c6daeb12ba9fa5340a4e4841b741e0499c8a337985fafaf Sep 09 10:48:12.956160 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:12.956127 2559 generic.go:356] "Generic (PLEG): container finished" podID="ddeb674b-b9af-40e9-861b-0aa0a3f216c4" containerID="7d4cab799ba5f7a431b2c38796ff7c09063a6775b931cd7ebd2ff49339d9f160" exitCode=0 Sep 09 10:48:12.956545 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:12.956211 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40f1e9-knhnp" event={"ID":"ddeb674b-b9af-40e9-861b-0aa0a3f216c4","Type":"ContainerDied","Data":"7d4cab799ba5f7a431b2c38796ff7c09063a6775b931cd7ebd2ff49339d9f160"} Sep 09 10:48:12.956545 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:12.956268 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40f1e9-knhnp" event={"ID":"ddeb674b-b9af-40e9-861b-0aa0a3f216c4","Type":"ContainerStarted","Data":"30bdb450b21ef03d1c6daeb12ba9fa5340a4e4841b741e0499c8a337985fafaf"} Sep 09 10:48:14.098217 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.098185 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40f1e9-knhnp" Sep 09 10:48:14.116723 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.116690 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddeb674b-b9af-40e9-861b-0aa0a3f216c4-maas-api-service-ca\") pod \"ddeb674b-b9af-40e9-861b-0aa0a3f216c4\" (UID: \"ddeb674b-b9af-40e9-861b-0aa0a3f216c4\") " Sep 09 10:48:14.117066 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.117039 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddeb674b-b9af-40e9-861b-0aa0a3f216c4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ddeb674b-b9af-40e9-861b-0aa0a3f216c4" (UID: "ddeb674b-b9af-40e9-861b-0aa0a3f216c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:48:14.117179 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.117163 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddeb674b-b9af-40e9-861b-0aa0a3f216c4-maas-api-service-ca\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:48:14.567437 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.567400 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-928d2a-57cd5d8fc7-wclgv"] Sep 09 10:48:14.567684 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.567654 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-928d2a-57cd5d8fc7-wclgv" podUID="89f95a27-8b8f-4902-8092-f7dd8744cd82" containerName="payload-processing" containerID="cri-o://35cd6f4a56482a1fe6bad98c914229333c01061626957af046eef5e974e5b156" gracePeriod=30 Sep 09 10:48:14.582190 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.582153 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-928d2a-68487959c8-t86mr"] Sep 09 10:48:14.582816 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.582712 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-928d2a-68487959c8-t86mr" podUID="bb3c8fef-1358-4580-8bbb-c87af9d7500e" containerName="payload-pre-processing" containerID="cri-o://9d8a9d04f03e259e1c518a7ceb0cc47c7b583d176a056d2c821255c31756817d" gracePeriod=30 Sep 09 10:48:14.751218 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.751148 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-928d2a-9p2r2"] Sep 09 10:48:14.753943 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.753909 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-928d2a-9p2r2"] Sep 09 10:48:14.851047 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.851023 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-928d2a-57cd5d8fc7-wclgv" Sep 09 10:48:14.858353 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.858332 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-928d2a-68487959c8-t86mr" Sep 09 10:48:14.924807 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.924726 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb3c8fef-1358-4580-8bbb-c87af9d7500e-plugins-config-volume\") pod \"bb3c8fef-1358-4580-8bbb-c87af9d7500e\" (UID: \"bb3c8fef-1358-4580-8bbb-c87af9d7500e\") " Sep 09 10:48:14.924967 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.924820 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8z6xh\" (UniqueName: \"kubernetes.io/projected/bb3c8fef-1358-4580-8bbb-c87af9d7500e-kube-api-access-8z6xh\") pod \"bb3c8fef-1358-4580-8bbb-c87af9d7500e\" (UID: \"bb3c8fef-1358-4580-8bbb-c87af9d7500e\") " Sep 09 10:48:14.924967 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.924852 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjl6l\" (UniqueName: \"kubernetes.io/projected/89f95a27-8b8f-4902-8092-f7dd8744cd82-kube-api-access-mjl6l\") pod \"89f95a27-8b8f-4902-8092-f7dd8744cd82\" (UID: \"89f95a27-8b8f-4902-8092-f7dd8744cd82\") " Sep 09 10:48:14.924967 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.924872 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89f95a27-8b8f-4902-8092-f7dd8744cd82-plugins-config-volume\") pod \"89f95a27-8b8f-4902-8092-f7dd8744cd82\" (UID: \"89f95a27-8b8f-4902-8092-f7dd8744cd82\") " Sep 09 10:48:14.925278 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.925247 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3c8fef-1358-4580-8bbb-c87af9d7500e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bb3c8fef-1358-4580-8bbb-c87af9d7500e" (UID: "bb3c8fef-1358-4580-8bbb-c87af9d7500e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:48:14.925278 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.925261 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89f95a27-8b8f-4902-8092-f7dd8744cd82-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "89f95a27-8b8f-4902-8092-f7dd8744cd82" (UID: "89f95a27-8b8f-4902-8092-f7dd8744cd82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:48:14.927101 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.927074 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f95a27-8b8f-4902-8092-f7dd8744cd82-kube-api-access-mjl6l" (OuterVolumeSpecName: "kube-api-access-mjl6l") pod "89f95a27-8b8f-4902-8092-f7dd8744cd82" (UID: "89f95a27-8b8f-4902-8092-f7dd8744cd82"). InnerVolumeSpecName "kube-api-access-mjl6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:48:14.927101 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.927092 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3c8fef-1358-4580-8bbb-c87af9d7500e-kube-api-access-8z6xh" (OuterVolumeSpecName: "kube-api-access-8z6xh") pod "bb3c8fef-1358-4580-8bbb-c87af9d7500e" (UID: "bb3c8fef-1358-4580-8bbb-c87af9d7500e"). InnerVolumeSpecName "kube-api-access-8z6xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:48:14.966751 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.966718 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40f1e9-knhnp" Sep 09 10:48:14.966751 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.966744 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40f1e9-knhnp" event={"ID":"ddeb674b-b9af-40e9-861b-0aa0a3f216c4","Type":"ContainerDied","Data":"30bdb450b21ef03d1c6daeb12ba9fa5340a4e4841b741e0499c8a337985fafaf"} Sep 09 10:48:14.967004 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.966792 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30bdb450b21ef03d1c6daeb12ba9fa5340a4e4841b741e0499c8a337985fafaf" Sep 09 10:48:14.968062 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.968034 2559 generic.go:356] "Generic (PLEG): container finished" podID="bb3c8fef-1358-4580-8bbb-c87af9d7500e" containerID="9d8a9d04f03e259e1c518a7ceb0cc47c7b583d176a056d2c821255c31756817d" exitCode=0 Sep 09 10:48:14.968176 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.968071 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-928d2a-68487959c8-t86mr" Sep 09 10:48:14.968176 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.968120 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-928d2a-68487959c8-t86mr" event={"ID":"bb3c8fef-1358-4580-8bbb-c87af9d7500e","Type":"ContainerDied","Data":"9d8a9d04f03e259e1c518a7ceb0cc47c7b583d176a056d2c821255c31756817d"} Sep 09 10:48:14.968281 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.968179 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-928d2a-68487959c8-t86mr" event={"ID":"bb3c8fef-1358-4580-8bbb-c87af9d7500e","Type":"ContainerDied","Data":"0233d55c7dabff5276928092cd3fa078ad605880c868bd89daba680a326552b4"} Sep 09 10:48:14.968281 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.968210 2559 scope.go:117] "RemoveContainer" containerID="9d8a9d04f03e259e1c518a7ceb0cc47c7b583d176a056d2c821255c31756817d" Sep 09 10:48:14.969978 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.969658 2559 generic.go:356] "Generic (PLEG): container finished" podID="89f95a27-8b8f-4902-8092-f7dd8744cd82" containerID="35cd6f4a56482a1fe6bad98c914229333c01061626957af046eef5e974e5b156" exitCode=0 Sep 09 10:48:14.969978 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.969707 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-928d2a-57cd5d8fc7-wclgv" Sep 09 10:48:14.969978 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.969735 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-928d2a-57cd5d8fc7-wclgv" event={"ID":"89f95a27-8b8f-4902-8092-f7dd8744cd82","Type":"ContainerDied","Data":"35cd6f4a56482a1fe6bad98c914229333c01061626957af046eef5e974e5b156"} Sep 09 10:48:14.969978 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.969788 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-928d2a-57cd5d8fc7-wclgv" event={"ID":"89f95a27-8b8f-4902-8092-f7dd8744cd82","Type":"ContainerDied","Data":"7a07033acea1eea3ae520498827a9b8ea041b484477db291fb3e62d9f99f86b9"} Sep 09 10:48:14.978050 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.978028 2559 scope.go:117] "RemoveContainer" containerID="9d8a9d04f03e259e1c518a7ceb0cc47c7b583d176a056d2c821255c31756817d" Sep 09 10:48:14.978387 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:48:14.978362 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d8a9d04f03e259e1c518a7ceb0cc47c7b583d176a056d2c821255c31756817d\": container with ID starting with 9d8a9d04f03e259e1c518a7ceb0cc47c7b583d176a056d2c821255c31756817d not found: ID does not exist" containerID="9d8a9d04f03e259e1c518a7ceb0cc47c7b583d176a056d2c821255c31756817d" Sep 09 10:48:14.978458 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.978397 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d8a9d04f03e259e1c518a7ceb0cc47c7b583d176a056d2c821255c31756817d"} err="failed to get container status \"9d8a9d04f03e259e1c518a7ceb0cc47c7b583d176a056d2c821255c31756817d\": rpc error: code = NotFound desc = could not find container \"9d8a9d04f03e259e1c518a7ceb0cc47c7b583d176a056d2c821255c31756817d\": container with ID starting with 9d8a9d04f03e259e1c518a7ceb0cc47c7b583d176a056d2c821255c31756817d not found: ID does not exist" Sep 09 10:48:14.978458 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.978422 2559 scope.go:117] "RemoveContainer" containerID="35cd6f4a56482a1fe6bad98c914229333c01061626957af046eef5e974e5b156" Sep 09 10:48:14.989106 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.989087 2559 scope.go:117] "RemoveContainer" containerID="35cd6f4a56482a1fe6bad98c914229333c01061626957af046eef5e974e5b156" Sep 09 10:48:14.989599 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:48:14.989577 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35cd6f4a56482a1fe6bad98c914229333c01061626957af046eef5e974e5b156\": container with ID starting with 35cd6f4a56482a1fe6bad98c914229333c01061626957af046eef5e974e5b156 not found: ID does not exist" containerID="35cd6f4a56482a1fe6bad98c914229333c01061626957af046eef5e974e5b156" Sep 09 10:48:14.989652 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.989608 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35cd6f4a56482a1fe6bad98c914229333c01061626957af046eef5e974e5b156"} err="failed to get container status \"35cd6f4a56482a1fe6bad98c914229333c01061626957af046eef5e974e5b156\": rpc error: code = NotFound desc = could not find container \"35cd6f4a56482a1fe6bad98c914229333c01061626957af046eef5e974e5b156\": container with ID starting with 35cd6f4a56482a1fe6bad98c914229333c01061626957af046eef5e974e5b156 not found: ID does not exist" Sep 09 10:48:14.994868 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.994845 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-928d2a-57cd5d8fc7-wclgv"] Sep 09 10:48:14.999951 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:14.999930 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-928d2a-57cd5d8fc7-wclgv"] Sep 09 10:48:15.011045 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:15.011022 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-928d2a-68487959c8-t86mr"] Sep 09 10:48:15.015197 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:15.015174 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-928d2a-68487959c8-t86mr"] Sep 09 10:48:15.025744 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:15.025724 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89f95a27-8b8f-4902-8092-f7dd8744cd82-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:48:15.025823 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:15.025745 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb3c8fef-1358-4580-8bbb-c87af9d7500e-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:48:15.025823 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:15.025756 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z6xh\" (UniqueName: \"kubernetes.io/projected/bb3c8fef-1358-4580-8bbb-c87af9d7500e-kube-api-access-8z6xh\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:48:15.025823 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:15.025765 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjl6l\" (UniqueName: \"kubernetes.io/projected/89f95a27-8b8f-4902-8092-f7dd8744cd82-kube-api-access-mjl6l\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:48:16.333903 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:16.333867 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="144fca0b-75fb-43fe-9c89-1be4e5283bf4" path="/var/lib/kubelet/pods/144fca0b-75fb-43fe-9c89-1be4e5283bf4/volumes" Sep 09 10:48:16.334274 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:16.334211 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89f95a27-8b8f-4902-8092-f7dd8744cd82" path="/var/lib/kubelet/pods/89f95a27-8b8f-4902-8092-f7dd8744cd82/volumes" Sep 09 10:48:16.334546 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:16.334530 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb3c8fef-1358-4580-8bbb-c87af9d7500e" path="/var/lib/kubelet/pods/bb3c8fef-1358-4580-8bbb-c87af9d7500e/volumes" Sep 09 10:48:22.394622 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:22.394531 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-40f1e9-56bfcc87ff-9j7vl"] Sep 09 10:48:22.395012 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:22.394788 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-40f1e9-56bfcc87ff-9j7vl" podUID="ffc08e84-a6f3-4d36-8969-c83a498924aa" containerName="payload-processing" containerID="cri-o://c6f016ec366c6decf22fb9b5bfce90f34642f2d3a0475ee5f3c70af7241c5731" gracePeriod=30 Sep 09 10:48:22.402068 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:22.402035 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-40f1e9-5d4c4cf546-qvvtb"] Sep 09 10:48:22.402276 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:22.402241 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40f1e9-5d4c4cf546-qvvtb" podUID="b698c646-6e14-4275-bc2f-3d731843a697" containerName="payload-pre-processing" containerID="cri-o://32c6f4de2297096d3c6afd84d29f3b55968706bf1e276ae1a5f72f313b66faed" gracePeriod=30 Sep 09 10:48:22.591675 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:22.591645 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40f1e9-knhnp"] Sep 09 10:48:22.597063 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:22.597039 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-40f1e9-knhnp"] Sep 09 10:48:22.682067 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:22.682044 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-40f1e9-56bfcc87ff-9j7vl" Sep 09 10:48:22.684544 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:22.684526 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40f1e9-5d4c4cf546-qvvtb" Sep 09 10:48:22.793333 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:22.793298 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b698c646-6e14-4275-bc2f-3d731843a697-plugins-config-volume\") pod \"b698c646-6e14-4275-bc2f-3d731843a697\" (UID: \"b698c646-6e14-4275-bc2f-3d731843a697\") " Sep 09 10:48:22.793333 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:22.793340 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffc08e84-a6f3-4d36-8969-c83a498924aa-plugins-config-volume\") pod \"ffc08e84-a6f3-4d36-8969-c83a498924aa\" (UID: \"ffc08e84-a6f3-4d36-8969-c83a498924aa\") " Sep 09 10:48:22.793600 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:22.793404 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t66vm\" (UniqueName: \"kubernetes.io/projected/ffc08e84-a6f3-4d36-8969-c83a498924aa-kube-api-access-t66vm\") pod \"ffc08e84-a6f3-4d36-8969-c83a498924aa\" (UID: \"ffc08e84-a6f3-4d36-8969-c83a498924aa\") " Sep 09 10:48:22.793600 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:22.793424 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcgmr\" (UniqueName: \"kubernetes.io/projected/b698c646-6e14-4275-bc2f-3d731843a697-kube-api-access-qcgmr\") pod \"b698c646-6e14-4275-bc2f-3d731843a697\" (UID: \"b698c646-6e14-4275-bc2f-3d731843a697\") " Sep 09 10:48:22.793767 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:22.793738 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b698c646-6e14-4275-bc2f-3d731843a697-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b698c646-6e14-4275-bc2f-3d731843a697" (UID: "b698c646-6e14-4275-bc2f-3d731843a697"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:48:22.793835 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:22.793764 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc08e84-a6f3-4d36-8969-c83a498924aa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ffc08e84-a6f3-4d36-8969-c83a498924aa" (UID: "ffc08e84-a6f3-4d36-8969-c83a498924aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:48:22.795662 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:22.795640 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b698c646-6e14-4275-bc2f-3d731843a697-kube-api-access-qcgmr" (OuterVolumeSpecName: "kube-api-access-qcgmr") pod "b698c646-6e14-4275-bc2f-3d731843a697" (UID: "b698c646-6e14-4275-bc2f-3d731843a697"). InnerVolumeSpecName "kube-api-access-qcgmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:48:22.795722 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:22.795670 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc08e84-a6f3-4d36-8969-c83a498924aa-kube-api-access-t66vm" (OuterVolumeSpecName: "kube-api-access-t66vm") pod "ffc08e84-a6f3-4d36-8969-c83a498924aa" (UID: "ffc08e84-a6f3-4d36-8969-c83a498924aa"). InnerVolumeSpecName "kube-api-access-t66vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:48:22.894415 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:22.894378 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t66vm\" (UniqueName: \"kubernetes.io/projected/ffc08e84-a6f3-4d36-8969-c83a498924aa-kube-api-access-t66vm\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:48:22.894415 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:22.894409 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcgmr\" (UniqueName: \"kubernetes.io/projected/b698c646-6e14-4275-bc2f-3d731843a697-kube-api-access-qcgmr\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:48:22.894415 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:22.894422 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b698c646-6e14-4275-bc2f-3d731843a697-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:48:22.894671 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:22.894430 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffc08e84-a6f3-4d36-8969-c83a498924aa-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:48:23.025748 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:23.025665 2559 generic.go:356] "Generic (PLEG): container finished" podID="ffc08e84-a6f3-4d36-8969-c83a498924aa" containerID="c6f016ec366c6decf22fb9b5bfce90f34642f2d3a0475ee5f3c70af7241c5731" exitCode=0 Sep 09 10:48:23.025748 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:23.025711 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-40f1e9-56bfcc87ff-9j7vl" Sep 09 10:48:23.025949 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:23.025745 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-40f1e9-56bfcc87ff-9j7vl" event={"ID":"ffc08e84-a6f3-4d36-8969-c83a498924aa","Type":"ContainerDied","Data":"c6f016ec366c6decf22fb9b5bfce90f34642f2d3a0475ee5f3c70af7241c5731"} Sep 09 10:48:23.025949 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:23.025801 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-40f1e9-56bfcc87ff-9j7vl" event={"ID":"ffc08e84-a6f3-4d36-8969-c83a498924aa","Type":"ContainerDied","Data":"896f9600c61ea7e3c7836a049ec2a31e33d4a448174fc637421f70bad8bb2a01"} Sep 09 10:48:23.025949 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:23.025831 2559 scope.go:117] "RemoveContainer" containerID="c6f016ec366c6decf22fb9b5bfce90f34642f2d3a0475ee5f3c70af7241c5731" Sep 09 10:48:23.027232 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:23.027210 2559 generic.go:356] "Generic (PLEG): container finished" podID="b698c646-6e14-4275-bc2f-3d731843a697" containerID="32c6f4de2297096d3c6afd84d29f3b55968706bf1e276ae1a5f72f313b66faed" exitCode=0 Sep 09 10:48:23.027339 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:23.027242 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40f1e9-5d4c4cf546-qvvtb" Sep 09 10:48:23.027339 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:23.027296 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40f1e9-5d4c4cf546-qvvtb" event={"ID":"b698c646-6e14-4275-bc2f-3d731843a697","Type":"ContainerDied","Data":"32c6f4de2297096d3c6afd84d29f3b55968706bf1e276ae1a5f72f313b66faed"} Sep 09 10:48:23.027454 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:23.027350 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-40f1e9-5d4c4cf546-qvvtb" event={"ID":"b698c646-6e14-4275-bc2f-3d731843a697","Type":"ContainerDied","Data":"d9b6fb0610db5193f2eecaf1f52024b6d5967cd7477c1c47aeb6d8d9b0fd7418"} Sep 09 10:48:23.036177 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:23.036159 2559 scope.go:117] "RemoveContainer" containerID="c6f016ec366c6decf22fb9b5bfce90f34642f2d3a0475ee5f3c70af7241c5731" Sep 09 10:48:23.036465 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:48:23.036444 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6f016ec366c6decf22fb9b5bfce90f34642f2d3a0475ee5f3c70af7241c5731\": container with ID starting with c6f016ec366c6decf22fb9b5bfce90f34642f2d3a0475ee5f3c70af7241c5731 not found: ID does not exist" containerID="c6f016ec366c6decf22fb9b5bfce90f34642f2d3a0475ee5f3c70af7241c5731" Sep 09 10:48:23.036568 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:23.036472 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6f016ec366c6decf22fb9b5bfce90f34642f2d3a0475ee5f3c70af7241c5731"} err="failed to get container status \"c6f016ec366c6decf22fb9b5bfce90f34642f2d3a0475ee5f3c70af7241c5731\": rpc error: code = NotFound desc = could not find container \"c6f016ec366c6decf22fb9b5bfce90f34642f2d3a0475ee5f3c70af7241c5731\": container with ID starting with c6f016ec366c6decf22fb9b5bfce90f34642f2d3a0475ee5f3c70af7241c5731 not found: ID does not exist" Sep 09 10:48:23.036568 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:23.036488 2559 scope.go:117] "RemoveContainer" containerID="32c6f4de2297096d3c6afd84d29f3b55968706bf1e276ae1a5f72f313b66faed" Sep 09 10:48:23.044862 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:23.044843 2559 scope.go:117] "RemoveContainer" containerID="32c6f4de2297096d3c6afd84d29f3b55968706bf1e276ae1a5f72f313b66faed" Sep 09 10:48:23.045185 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:48:23.045168 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32c6f4de2297096d3c6afd84d29f3b55968706bf1e276ae1a5f72f313b66faed\": container with ID starting with 32c6f4de2297096d3c6afd84d29f3b55968706bf1e276ae1a5f72f313b66faed not found: ID does not exist" containerID="32c6f4de2297096d3c6afd84d29f3b55968706bf1e276ae1a5f72f313b66faed" Sep 09 10:48:23.045247 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:23.045192 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32c6f4de2297096d3c6afd84d29f3b55968706bf1e276ae1a5f72f313b66faed"} err="failed to get container status \"32c6f4de2297096d3c6afd84d29f3b55968706bf1e276ae1a5f72f313b66faed\": rpc error: code = NotFound desc = could not find container \"32c6f4de2297096d3c6afd84d29f3b55968706bf1e276ae1a5f72f313b66faed\": container with ID starting with 32c6f4de2297096d3c6afd84d29f3b55968706bf1e276ae1a5f72f313b66faed not found: ID does not exist" Sep 09 10:48:23.048603 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:23.048579 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-40f1e9-56bfcc87ff-9j7vl"] Sep 09 10:48:23.056598 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:23.056576 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-40f1e9-56bfcc87ff-9j7vl"] Sep 09 10:48:23.067544 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:23.067524 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-40f1e9-5d4c4cf546-qvvtb"] Sep 09 10:48:23.073985 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:23.073966 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-40f1e9-5d4c4cf546-qvvtb"] Sep 09 10:48:24.333692 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:24.333657 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b698c646-6e14-4275-bc2f-3d731843a697" path="/var/lib/kubelet/pods/b698c646-6e14-4275-bc2f-3d731843a697/volumes" Sep 09 10:48:24.334190 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:24.334125 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddeb674b-b9af-40e9-861b-0aa0a3f216c4" path="/var/lib/kubelet/pods/ddeb674b-b9af-40e9-861b-0aa0a3f216c4/volumes" Sep 09 10:48:24.334515 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:24.334482 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffc08e84-a6f3-4d36-8969-c83a498924aa" path="/var/lib/kubelet/pods/ffc08e84-a6f3-4d36-8969-c83a498924aa/volumes" Sep 09 10:48:29.987673 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:29.987636 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da3c64-drrtl"] Sep 09 10:48:29.988029 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:29.987990 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b698c646-6e14-4275-bc2f-3d731843a697" containerName="payload-pre-processing" Sep 09 10:48:29.988029 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:29.988002 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b698c646-6e14-4275-bc2f-3d731843a697" containerName="payload-pre-processing" Sep 09 10:48:29.988029 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:29.988023 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffc08e84-a6f3-4d36-8969-c83a498924aa" containerName="payload-processing" Sep 09 10:48:29.988136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:29.988033 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffc08e84-a6f3-4d36-8969-c83a498924aa" containerName="payload-processing" Sep 09 10:48:29.988136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:29.988040 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddeb674b-b9af-40e9-861b-0aa0a3f216c4" containerName="revoke-keys" Sep 09 10:48:29.988136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:29.988046 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddeb674b-b9af-40e9-861b-0aa0a3f216c4" containerName="revoke-keys" Sep 09 10:48:29.988136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:29.988053 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89f95a27-8b8f-4902-8092-f7dd8744cd82" containerName="payload-processing" Sep 09 10:48:29.988136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:29.988059 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="89f95a27-8b8f-4902-8092-f7dd8744cd82" containerName="payload-processing" Sep 09 10:48:29.988136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:29.988065 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb3c8fef-1358-4580-8bbb-c87af9d7500e" containerName="payload-pre-processing" Sep 09 10:48:29.988136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:29.988070 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb3c8fef-1358-4580-8bbb-c87af9d7500e" containerName="payload-pre-processing" Sep 09 10:48:29.988136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:29.988136 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddeb674b-b9af-40e9-861b-0aa0a3f216c4" containerName="revoke-keys" Sep 09 10:48:29.988366 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:29.988146 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffc08e84-a6f3-4d36-8969-c83a498924aa" containerName="payload-processing" Sep 09 10:48:29.988366 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:29.988152 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb3c8fef-1358-4580-8bbb-c87af9d7500e" containerName="payload-pre-processing" Sep 09 10:48:29.988366 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:29.988158 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b698c646-6e14-4275-bc2f-3d731843a697" containerName="payload-pre-processing" Sep 09 10:48:29.988366 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:29.988164 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="89f95a27-8b8f-4902-8092-f7dd8744cd82" containerName="payload-processing" Sep 09 10:48:29.991080 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:29.991063 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da3c64-drrtl" Sep 09 10:48:29.993810 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:29.993785 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kq7qv\"" Sep 09 10:48:29.993810 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:29.993805 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 10:48:30.000386 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:30.000362 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da3c64-drrtl"] Sep 09 10:48:30.053315 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:30.053264 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4105506f-c43c-4445-90dc-7988d84c44d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-da3c64-drrtl\" (UID: \"4105506f-c43c-4445-90dc-7988d84c44d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da3c64-drrtl" Sep 09 10:48:30.153899 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:30.153863 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4105506f-c43c-4445-90dc-7988d84c44d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-da3c64-drrtl\" (UID: \"4105506f-c43c-4445-90dc-7988d84c44d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da3c64-drrtl" Sep 09 10:48:30.154468 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:30.154447 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4105506f-c43c-4445-90dc-7988d84c44d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-da3c64-drrtl\" (UID: \"4105506f-c43c-4445-90dc-7988d84c44d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da3c64-drrtl" Sep 09 10:48:30.303112 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:30.303027 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da3c64-drrtl" Sep 09 10:48:30.578332 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:30.578309 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da3c64-drrtl"] Sep 09 10:48:30.580647 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:48:30.580622 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4105506f_c43c_4445_90dc_7988d84c44d9.slice/crio-9d2df6c7f6ee2250687d75af73809d2145b918309c7b5d9f0ef627c237b721ab WatchSource:0}: Error finding container 9d2df6c7f6ee2250687d75af73809d2145b918309c7b5d9f0ef627c237b721ab: Status 404 returned error can't find the container with id 9d2df6c7f6ee2250687d75af73809d2145b918309c7b5d9f0ef627c237b721ab Sep 09 10:48:31.070783 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:31.070739 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da3c64-drrtl" event={"ID":"4105506f-c43c-4445-90dc-7988d84c44d9","Type":"ContainerStarted","Data":"d55c998bcde81ff56f841f3e715f799414be739ac20812f7f9660ec3291488c0"} Sep 09 10:48:31.071139 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:31.070794 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da3c64-drrtl" event={"ID":"4105506f-c43c-4445-90dc-7988d84c44d9","Type":"ContainerStarted","Data":"9d2df6c7f6ee2250687d75af73809d2145b918309c7b5d9f0ef627c237b721ab"} Sep 09 10:48:31.086991 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:31.086941 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da3c64-drrtl" podStartSLOduration=2.086925766 podStartE2EDuration="2.086925766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:48:31.085610505 +0000 UTC m=+1351.260212853" watchObservedRunningTime="2026-09-09 10:48:31.086925766 +0000 UTC m=+1351.261528090" Sep 09 10:48:32.077952 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:32.077916 2559 generic.go:356] "Generic (PLEG): container finished" podID="4105506f-c43c-4445-90dc-7988d84c44d9" containerID="d55c998bcde81ff56f841f3e715f799414be739ac20812f7f9660ec3291488c0" exitCode=0 Sep 09 10:48:32.078341 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:32.077999 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da3c64-drrtl" event={"ID":"4105506f-c43c-4445-90dc-7988d84c44d9","Type":"ContainerDied","Data":"d55c998bcde81ff56f841f3e715f799414be739ac20812f7f9660ec3291488c0"} Sep 09 10:48:33.220159 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:33.220132 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da3c64-drrtl" Sep 09 10:48:33.279700 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:33.279657 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4105506f-c43c-4445-90dc-7988d84c44d9-maas-api-service-ca\") pod \"4105506f-c43c-4445-90dc-7988d84c44d9\" (UID: \"4105506f-c43c-4445-90dc-7988d84c44d9\") " Sep 09 10:48:33.280050 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:33.280025 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4105506f-c43c-4445-90dc-7988d84c44d9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4105506f-c43c-4445-90dc-7988d84c44d9" (UID: "4105506f-c43c-4445-90dc-7988d84c44d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:48:33.380308 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:33.380224 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4105506f-c43c-4445-90dc-7988d84c44d9-maas-api-service-ca\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:48:34.089733 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:34.089702 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da3c64-drrtl" Sep 09 10:48:34.089886 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:34.089733 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da3c64-drrtl" event={"ID":"4105506f-c43c-4445-90dc-7988d84c44d9","Type":"ContainerDied","Data":"9d2df6c7f6ee2250687d75af73809d2145b918309c7b5d9f0ef627c237b721ab"} Sep 09 10:48:34.089886 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:34.089777 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d2df6c7f6ee2250687d75af73809d2145b918309c7b5d9f0ef627c237b721ab" Sep 09 10:48:40.605435 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:40.605399 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-da3c64-78894768f-xfdwn"] Sep 09 10:48:40.606525 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:40.606451 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-da3c64-78894768f-xfdwn" podUID="9bbb87f1-9446-4f65-a9aa-85704da050e9" containerName="payload-processing" containerID="cri-o://c5f8c568a3f4dc94b25896fad3421f7798d3a5908f578e65cc653ee949b26f1f" gracePeriod=30 Sep 09 10:48:40.609951 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:40.609928 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-da3c64-644699bc86-86wjk"] Sep 09 10:48:40.610616 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:40.610571 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da3c64-644699bc86-86wjk" podUID="d7218d2e-5072-4134-a5d4-e5865febfe64" containerName="payload-pre-processing" containerID="cri-o://52d06511ba69f1cec744aa5ba9fe069f7e6ebedd5c27077b6a78195b546364e7" gracePeriod=30 Sep 09 10:48:40.781285 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:40.781215 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da3c64-drrtl"] Sep 09 10:48:40.785228 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:40.785196 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da3c64-drrtl"] Sep 09 10:48:40.906457 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:40.906421 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-da3c64-78894768f-xfdwn" Sep 09 10:48:40.909200 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:40.909178 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da3c64-644699bc86-86wjk" Sep 09 10:48:41.052962 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:41.052907 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6nds\" (UniqueName: \"kubernetes.io/projected/9bbb87f1-9446-4f65-a9aa-85704da050e9-kube-api-access-s6nds\") pod \"9bbb87f1-9446-4f65-a9aa-85704da050e9\" (UID: \"9bbb87f1-9446-4f65-a9aa-85704da050e9\") " Sep 09 10:48:41.053136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:41.053019 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7218d2e-5072-4134-a5d4-e5865febfe64-plugins-config-volume\") pod \"d7218d2e-5072-4134-a5d4-e5865febfe64\" (UID: \"d7218d2e-5072-4134-a5d4-e5865febfe64\") " Sep 09 10:48:41.053136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:41.053062 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bbb87f1-9446-4f65-a9aa-85704da050e9-plugins-config-volume\") pod \"9bbb87f1-9446-4f65-a9aa-85704da050e9\" (UID: \"9bbb87f1-9446-4f65-a9aa-85704da050e9\") " Sep 09 10:48:41.053136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:41.053093 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bkbn\" (UniqueName: \"kubernetes.io/projected/d7218d2e-5072-4134-a5d4-e5865febfe64-kube-api-access-6bkbn\") pod \"d7218d2e-5072-4134-a5d4-e5865febfe64\" (UID: \"d7218d2e-5072-4134-a5d4-e5865febfe64\") " Sep 09 10:48:41.053407 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:41.053376 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7218d2e-5072-4134-a5d4-e5865febfe64-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d7218d2e-5072-4134-a5d4-e5865febfe64" (UID: "d7218d2e-5072-4134-a5d4-e5865febfe64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:48:41.053486 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:41.053413 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bbb87f1-9446-4f65-a9aa-85704da050e9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9bbb87f1-9446-4f65-a9aa-85704da050e9" (UID: "9bbb87f1-9446-4f65-a9aa-85704da050e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:48:41.055223 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:41.055199 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bbb87f1-9446-4f65-a9aa-85704da050e9-kube-api-access-s6nds" (OuterVolumeSpecName: "kube-api-access-s6nds") pod "9bbb87f1-9446-4f65-a9aa-85704da050e9" (UID: "9bbb87f1-9446-4f65-a9aa-85704da050e9"). InnerVolumeSpecName "kube-api-access-s6nds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:48:41.055344 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:41.055249 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7218d2e-5072-4134-a5d4-e5865febfe64-kube-api-access-6bkbn" (OuterVolumeSpecName: "kube-api-access-6bkbn") pod "d7218d2e-5072-4134-a5d4-e5865febfe64" (UID: "d7218d2e-5072-4134-a5d4-e5865febfe64"). InnerVolumeSpecName "kube-api-access-6bkbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:48:41.127324 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:41.127283 2559 generic.go:356] "Generic (PLEG): container finished" podID="9bbb87f1-9446-4f65-a9aa-85704da050e9" containerID="c5f8c568a3f4dc94b25896fad3421f7798d3a5908f578e65cc653ee949b26f1f" exitCode=0 Sep 09 10:48:41.127533 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:41.127332 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-da3c64-78894768f-xfdwn" Sep 09 10:48:41.127533 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:41.127370 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-da3c64-78894768f-xfdwn" event={"ID":"9bbb87f1-9446-4f65-a9aa-85704da050e9","Type":"ContainerDied","Data":"c5f8c568a3f4dc94b25896fad3421f7798d3a5908f578e65cc653ee949b26f1f"} Sep 09 10:48:41.127533 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:41.127420 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-da3c64-78894768f-xfdwn" event={"ID":"9bbb87f1-9446-4f65-a9aa-85704da050e9","Type":"ContainerDied","Data":"ecdfadbd23fa40787b0f5668251c738271850c5b21349851e0dda1fce4cfccb7"} Sep 09 10:48:41.127533 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:41.127448 2559 scope.go:117] "RemoveContainer" containerID="c5f8c568a3f4dc94b25896fad3421f7798d3a5908f578e65cc653ee949b26f1f" Sep 09 10:48:41.128929 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:41.128907 2559 generic.go:356] "Generic (PLEG): container finished" podID="d7218d2e-5072-4134-a5d4-e5865febfe64" containerID="52d06511ba69f1cec744aa5ba9fe069f7e6ebedd5c27077b6a78195b546364e7" exitCode=0 Sep 09 10:48:41.129022 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:41.128962 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da3c64-644699bc86-86wjk" Sep 09 10:48:41.129068 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:41.128962 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da3c64-644699bc86-86wjk" event={"ID":"d7218d2e-5072-4134-a5d4-e5865febfe64","Type":"ContainerDied","Data":"52d06511ba69f1cec744aa5ba9fe069f7e6ebedd5c27077b6a78195b546364e7"} Sep 09 10:48:41.129105 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:41.129088 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da3c64-644699bc86-86wjk" event={"ID":"d7218d2e-5072-4134-a5d4-e5865febfe64","Type":"ContainerDied","Data":"2f3f9a12bcd35cb9a19154afb9c67564ebf6555498bd16feeda8ca5bb66128f3"} Sep 09 10:48:41.139111 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:41.139087 2559 scope.go:117] "RemoveContainer" containerID="c5f8c568a3f4dc94b25896fad3421f7798d3a5908f578e65cc653ee949b26f1f" Sep 09 10:48:41.139457 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:48:41.139433 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5f8c568a3f4dc94b25896fad3421f7798d3a5908f578e65cc653ee949b26f1f\": container with ID starting with c5f8c568a3f4dc94b25896fad3421f7798d3a5908f578e65cc653ee949b26f1f not found: ID does not exist" containerID="c5f8c568a3f4dc94b25896fad3421f7798d3a5908f578e65cc653ee949b26f1f" Sep 09 10:48:41.139596 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:41.139470 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5f8c568a3f4dc94b25896fad3421f7798d3a5908f578e65cc653ee949b26f1f"} err="failed to get container status \"c5f8c568a3f4dc94b25896fad3421f7798d3a5908f578e65cc653ee949b26f1f\": rpc error: code = NotFound desc = could not find container \"c5f8c568a3f4dc94b25896fad3421f7798d3a5908f578e65cc653ee949b26f1f\": container with ID starting with c5f8c568a3f4dc94b25896fad3421f7798d3a5908f578e65cc653ee949b26f1f not found: ID does not exist" Sep 09 10:48:41.139596 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:41.139495 2559 scope.go:117] "RemoveContainer" containerID="52d06511ba69f1cec744aa5ba9fe069f7e6ebedd5c27077b6a78195b546364e7" Sep 09 10:48:41.148960 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:41.148940 2559 scope.go:117] "RemoveContainer" containerID="52d06511ba69f1cec744aa5ba9fe069f7e6ebedd5c27077b6a78195b546364e7" Sep 09 10:48:41.149255 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:48:41.149234 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52d06511ba69f1cec744aa5ba9fe069f7e6ebedd5c27077b6a78195b546364e7\": container with ID starting with 52d06511ba69f1cec744aa5ba9fe069f7e6ebedd5c27077b6a78195b546364e7 not found: ID does not exist" containerID="52d06511ba69f1cec744aa5ba9fe069f7e6ebedd5c27077b6a78195b546364e7" Sep 09 10:48:41.149353 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:41.149262 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52d06511ba69f1cec744aa5ba9fe069f7e6ebedd5c27077b6a78195b546364e7"} err="failed to get container status \"52d06511ba69f1cec744aa5ba9fe069f7e6ebedd5c27077b6a78195b546364e7\": rpc error: code = NotFound desc = could not find container \"52d06511ba69f1cec744aa5ba9fe069f7e6ebedd5c27077b6a78195b546364e7\": container with ID starting with 52d06511ba69f1cec744aa5ba9fe069f7e6ebedd5c27077b6a78195b546364e7 not found: ID does not exist" Sep 09 10:48:41.155296 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:41.155257 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7218d2e-5072-4134-a5d4-e5865febfe64-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:48:41.155296 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:41.155292 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bbb87f1-9446-4f65-a9aa-85704da050e9-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:48:41.155465 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:41.155315 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bkbn\" (UniqueName: \"kubernetes.io/projected/d7218d2e-5072-4134-a5d4-e5865febfe64-kube-api-access-6bkbn\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:48:41.155465 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:41.155329 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6nds\" (UniqueName: \"kubernetes.io/projected/9bbb87f1-9446-4f65-a9aa-85704da050e9-kube-api-access-s6nds\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:48:41.160873 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:41.160831 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-da3c64-78894768f-xfdwn"] Sep 09 10:48:41.162667 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:41.162644 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-da3c64-78894768f-xfdwn"] Sep 09 10:48:41.172131 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:41.172105 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-da3c64-644699bc86-86wjk"] Sep 09 10:48:41.175601 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:41.175582 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-da3c64-644699bc86-86wjk"] Sep 09 10:48:42.335271 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:42.335235 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4105506f-c43c-4445-90dc-7988d84c44d9" path="/var/lib/kubelet/pods/4105506f-c43c-4445-90dc-7988d84c44d9/volumes" Sep 09 10:48:42.335664 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:42.335619 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bbb87f1-9446-4f65-a9aa-85704da050e9" path="/var/lib/kubelet/pods/9bbb87f1-9446-4f65-a9aa-85704da050e9/volumes" Sep 09 10:48:42.335933 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:42.335920 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7218d2e-5072-4134-a5d4-e5865febfe64" path="/var/lib/kubelet/pods/d7218d2e-5072-4134-a5d4-e5865febfe64/volumes" Sep 09 10:48:53.746442 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:53.746403 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1eb21d-846bcbb58-p64nc"] Sep 09 10:48:53.746864 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:53.746767 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bbb87f1-9446-4f65-a9aa-85704da050e9" containerName="payload-processing" Sep 09 10:48:53.746864 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:53.746783 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bbb87f1-9446-4f65-a9aa-85704da050e9" containerName="payload-processing" Sep 09 10:48:53.746864 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:53.746808 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7218d2e-5072-4134-a5d4-e5865febfe64" containerName="payload-pre-processing" Sep 09 10:48:53.746864 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:53.746815 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7218d2e-5072-4134-a5d4-e5865febfe64" containerName="payload-pre-processing" Sep 09 10:48:53.746864 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:53.746825 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4105506f-c43c-4445-90dc-7988d84c44d9" containerName="revoke-keys" Sep 09 10:48:53.746864 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:53.746831 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4105506f-c43c-4445-90dc-7988d84c44d9" containerName="revoke-keys" Sep 09 10:48:53.747048 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:53.746879 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4105506f-c43c-4445-90dc-7988d84c44d9" containerName="revoke-keys" Sep 09 10:48:53.747048 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:53.746887 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7218d2e-5072-4134-a5d4-e5865febfe64" containerName="payload-pre-processing" Sep 09 10:48:53.747048 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:53.746894 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bbb87f1-9446-4f65-a9aa-85704da050e9" containerName="payload-processing" Sep 09 10:48:53.751249 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:53.751228 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1eb21d-846bcbb58-p64nc" Sep 09 10:48:53.755390 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:53.755364 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-1eb21d\"" Sep 09 10:48:53.755601 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:53.755577 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-1eb21d-dockercfg-gdccs\"" Sep 09 10:48:53.764321 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:53.764293 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1eb21d-846bcbb58-p64nc"] Sep 09 10:48:53.769689 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:53.769659 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1eb21d-57fcdc7b5-n5fr8"] Sep 09 10:48:53.772993 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:53.772972 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1eb21d-57fcdc7b5-n5fr8" Sep 09 10:48:53.790750 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:53.790706 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1eb21d-57fcdc7b5-n5fr8"] Sep 09 10:48:53.873439 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:53.873396 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc30260f-79d2-4307-ac41-0482974ad1e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1eb21d-846bcbb58-p64nc\" (UID: \"fc30260f-79d2-4307-ac41-0482974ad1e8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1eb21d-846bcbb58-p64nc" Sep 09 10:48:53.873640 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:53.873450 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1eb21d-57fcdc7b5-n5fr8\" (UID: \"39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1eb21d-57fcdc7b5-n5fr8" Sep 09 10:48:53.873640 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:53.873477 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb8rl\" (UniqueName: \"kubernetes.io/projected/39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d-kube-api-access-pb8rl\") pod \"payload-processing-e2e-shared-a-w3-1eb21d-57fcdc7b5-n5fr8\" (UID: \"39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1eb21d-57fcdc7b5-n5fr8" Sep 09 10:48:53.873640 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:53.873626 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlqz2\" (UniqueName: \"kubernetes.io/projected/fc30260f-79d2-4307-ac41-0482974ad1e8-kube-api-access-xlqz2\") pod \"payload-pre-processing-e2e-shared-a-w3-1eb21d-846bcbb58-p64nc\" (UID: \"fc30260f-79d2-4307-ac41-0482974ad1e8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1eb21d-846bcbb58-p64nc" Sep 09 10:48:53.974825 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:53.974778 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb8rl\" (UniqueName: \"kubernetes.io/projected/39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d-kube-api-access-pb8rl\") pod \"payload-processing-e2e-shared-a-w3-1eb21d-57fcdc7b5-n5fr8\" (UID: \"39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1eb21d-57fcdc7b5-n5fr8" Sep 09 10:48:53.975001 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:53.974842 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlqz2\" (UniqueName: \"kubernetes.io/projected/fc30260f-79d2-4307-ac41-0482974ad1e8-kube-api-access-xlqz2\") pod \"payload-pre-processing-e2e-shared-a-w3-1eb21d-846bcbb58-p64nc\" (UID: \"fc30260f-79d2-4307-ac41-0482974ad1e8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1eb21d-846bcbb58-p64nc" Sep 09 10:48:53.975001 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:53.974877 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc30260f-79d2-4307-ac41-0482974ad1e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1eb21d-846bcbb58-p64nc\" (UID: \"fc30260f-79d2-4307-ac41-0482974ad1e8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1eb21d-846bcbb58-p64nc" Sep 09 10:48:53.975001 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:53.974901 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1eb21d-57fcdc7b5-n5fr8\" (UID: \"39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1eb21d-57fcdc7b5-n5fr8" Sep 09 10:48:53.975680 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:53.975654 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc30260f-79d2-4307-ac41-0482974ad1e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1eb21d-846bcbb58-p64nc\" (UID: \"fc30260f-79d2-4307-ac41-0482974ad1e8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1eb21d-846bcbb58-p64nc" Sep 09 10:48:53.975861 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:53.975812 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1eb21d-57fcdc7b5-n5fr8\" (UID: \"39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1eb21d-57fcdc7b5-n5fr8" Sep 09 10:48:53.984338 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:53.984306 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb8rl\" (UniqueName: \"kubernetes.io/projected/39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d-kube-api-access-pb8rl\") pod \"payload-processing-e2e-shared-a-w3-1eb21d-57fcdc7b5-n5fr8\" (UID: \"39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1eb21d-57fcdc7b5-n5fr8" Sep 09 10:48:53.984465 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:53.984404 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlqz2\" (UniqueName: \"kubernetes.io/projected/fc30260f-79d2-4307-ac41-0482974ad1e8-kube-api-access-xlqz2\") pod \"payload-pre-processing-e2e-shared-a-w3-1eb21d-846bcbb58-p64nc\" (UID: \"fc30260f-79d2-4307-ac41-0482974ad1e8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1eb21d-846bcbb58-p64nc" Sep 09 10:48:54.063013 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:54.062922 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1eb21d-846bcbb58-p64nc" Sep 09 10:48:54.084587 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:54.084553 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1eb21d-57fcdc7b5-n5fr8" Sep 09 10:48:54.375874 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:54.375840 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1eb21d-846bcbb58-p64nc"] Sep 09 10:48:54.385330 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:48:54.385291 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc30260f_79d2_4307_ac41_0482974ad1e8.slice/crio-8a00fcbabce016b0cd6578fb9f3d6f543a566ea1b152d32620b2c45e39c4c069 WatchSource:0}: Error finding container 8a00fcbabce016b0cd6578fb9f3d6f543a566ea1b152d32620b2c45e39c4c069: Status 404 returned error can't find the container with id 8a00fcbabce016b0cd6578fb9f3d6f543a566ea1b152d32620b2c45e39c4c069 Sep 09 10:48:54.417645 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:54.417607 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1eb21d-57fcdc7b5-n5fr8"] Sep 09 10:48:54.421643 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:48:54.421609 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e7e1a2_ebc5_4a25_9be5_e4da4d7e2a0d.slice/crio-0dc33ab488666563ab8c873630712a3a5e9e3c696cf646d083b7162bf9188dd7 WatchSource:0}: Error finding container 0dc33ab488666563ab8c873630712a3a5e9e3c696cf646d083b7162bf9188dd7: Status 404 returned error can't find the container with id 0dc33ab488666563ab8c873630712a3a5e9e3c696cf646d083b7162bf9188dd7 Sep 09 10:48:55.203388 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:55.203345 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1eb21d-57fcdc7b5-n5fr8" event={"ID":"39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d","Type":"ContainerStarted","Data":"36bcc275cfa032b9507ec4d79f5daf100e1b86122072a7be3af9b82ef4d538ea"} Sep 09 10:48:55.203883 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:55.203422 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1eb21d-57fcdc7b5-n5fr8" Sep 09 10:48:55.203883 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:55.203450 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1eb21d-57fcdc7b5-n5fr8" event={"ID":"39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d","Type":"ContainerStarted","Data":"0dc33ab488666563ab8c873630712a3a5e9e3c696cf646d083b7162bf9188dd7"} Sep 09 10:48:55.204960 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:55.204937 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1eb21d-846bcbb58-p64nc" event={"ID":"fc30260f-79d2-4307-ac41-0482974ad1e8","Type":"ContainerStarted","Data":"abe3eb841370db13b923bad64284bcccef8b99ee7c3c26cee9693aec0e34c2e6"} Sep 09 10:48:55.205082 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:55.204979 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1eb21d-846bcbb58-p64nc" event={"ID":"fc30260f-79d2-4307-ac41-0482974ad1e8","Type":"ContainerStarted","Data":"8a00fcbabce016b0cd6578fb9f3d6f543a566ea1b152d32620b2c45e39c4c069"} Sep 09 10:48:55.205134 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:55.205079 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1eb21d-846bcbb58-p64nc" Sep 09 10:48:55.222479 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:55.222410 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1eb21d-57fcdc7b5-n5fr8" podStartSLOduration=2.222391488 podStartE2EDuration="2.222391488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:48:55.220731883 +0000 UTC m=+1375.395334210" watchObservedRunningTime="2026-09-09 10:48:55.222391488 +0000 UTC m=+1375.396993812" Sep 09 10:48:55.252957 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:55.252902 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1eb21d-846bcbb58-p64nc" podStartSLOduration=2.252881647 podStartE2EDuration="2.252881647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:48:55.249582094 +0000 UTC m=+1375.424184423" watchObservedRunningTime="2026-09-09 10:48:55.252881647 +0000 UTC m=+1375.427483972" Sep 09 10:48:56.258605 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:56.258544 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2a090f42-c644cb48b-8np6k"] Sep 09 10:48:56.262290 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:56.262265 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a090f42-c644cb48b-8np6k" Sep 09 10:48:56.265455 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:56.265432 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2a090f42\"" Sep 09 10:48:56.265630 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:56.265536 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2a090f42-dockercfg-v7nfp\"" Sep 09 10:48:56.273876 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:56.273843 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2a090f42-c644cb48b-8np6k"] Sep 09 10:48:56.285825 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:56.285788 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2a090f42-5d466c5697-q46lj"] Sep 09 10:48:56.289850 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:56.289821 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2a090f42-5d466c5697-q46lj" Sep 09 10:48:56.298188 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:56.298151 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2a090f42-5d466c5697-q46lj"] Sep 09 10:48:56.396543 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:56.396479 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06632265-dbd0-42f1-a9b5-7033ae63bcee-plugins-config-volume\") pod \"payload-processing-e2e-mt-2a090f42-5d466c5697-q46lj\" (UID: \"06632265-dbd0-42f1-a9b5-7033ae63bcee\") " pod="openshift-ingress/payload-processing-e2e-mt-2a090f42-5d466c5697-q46lj" Sep 09 10:48:56.396721 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:56.396645 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1866cfd9-c3e2-469a-baad-42bd6efce9c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2a090f42-c644cb48b-8np6k\" (UID: \"1866cfd9-c3e2-469a-baad-42bd6efce9c6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2a090f42-c644cb48b-8np6k" Sep 09 10:48:56.396721 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:56.396707 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrslq\" (UniqueName: \"kubernetes.io/projected/1866cfd9-c3e2-469a-baad-42bd6efce9c6-kube-api-access-qrslq\") pod \"payload-pre-processing-e2e-mt-2a090f42-c644cb48b-8np6k\" (UID: \"1866cfd9-c3e2-469a-baad-42bd6efce9c6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2a090f42-c644cb48b-8np6k" Sep 09 10:48:56.396806 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:56.396736 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpffw\" (UniqueName: \"kubernetes.io/projected/06632265-dbd0-42f1-a9b5-7033ae63bcee-kube-api-access-bpffw\") pod \"payload-processing-e2e-mt-2a090f42-5d466c5697-q46lj\" (UID: \"06632265-dbd0-42f1-a9b5-7033ae63bcee\") " pod="openshift-ingress/payload-processing-e2e-mt-2a090f42-5d466c5697-q46lj" Sep 09 10:48:56.497593 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:56.497558 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06632265-dbd0-42f1-a9b5-7033ae63bcee-plugins-config-volume\") pod \"payload-processing-e2e-mt-2a090f42-5d466c5697-q46lj\" (UID: \"06632265-dbd0-42f1-a9b5-7033ae63bcee\") " pod="openshift-ingress/payload-processing-e2e-mt-2a090f42-5d466c5697-q46lj" Sep 09 10:48:56.497769 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:56.497642 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1866cfd9-c3e2-469a-baad-42bd6efce9c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2a090f42-c644cb48b-8np6k\" (UID: \"1866cfd9-c3e2-469a-baad-42bd6efce9c6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2a090f42-c644cb48b-8np6k" Sep 09 10:48:56.497769 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:56.497673 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrslq\" (UniqueName: \"kubernetes.io/projected/1866cfd9-c3e2-469a-baad-42bd6efce9c6-kube-api-access-qrslq\") pod \"payload-pre-processing-e2e-mt-2a090f42-c644cb48b-8np6k\" (UID: \"1866cfd9-c3e2-469a-baad-42bd6efce9c6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2a090f42-c644cb48b-8np6k" Sep 09 10:48:56.497769 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:56.497688 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpffw\" (UniqueName: \"kubernetes.io/projected/06632265-dbd0-42f1-a9b5-7033ae63bcee-kube-api-access-bpffw\") pod \"payload-processing-e2e-mt-2a090f42-5d466c5697-q46lj\" (UID: \"06632265-dbd0-42f1-a9b5-7033ae63bcee\") " pod="openshift-ingress/payload-processing-e2e-mt-2a090f42-5d466c5697-q46lj" Sep 09 10:48:56.498325 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:56.498303 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06632265-dbd0-42f1-a9b5-7033ae63bcee-plugins-config-volume\") pod \"payload-processing-e2e-mt-2a090f42-5d466c5697-q46lj\" (UID: \"06632265-dbd0-42f1-a9b5-7033ae63bcee\") " pod="openshift-ingress/payload-processing-e2e-mt-2a090f42-5d466c5697-q46lj" Sep 09 10:48:56.498385 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:56.498366 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1866cfd9-c3e2-469a-baad-42bd6efce9c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2a090f42-c644cb48b-8np6k\" (UID: \"1866cfd9-c3e2-469a-baad-42bd6efce9c6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2a090f42-c644cb48b-8np6k" Sep 09 10:48:56.506867 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:56.506837 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrslq\" (UniqueName: \"kubernetes.io/projected/1866cfd9-c3e2-469a-baad-42bd6efce9c6-kube-api-access-qrslq\") pod \"payload-pre-processing-e2e-mt-2a090f42-c644cb48b-8np6k\" (UID: \"1866cfd9-c3e2-469a-baad-42bd6efce9c6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2a090f42-c644cb48b-8np6k" Sep 09 10:48:56.506966 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:56.506838 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpffw\" (UniqueName: \"kubernetes.io/projected/06632265-dbd0-42f1-a9b5-7033ae63bcee-kube-api-access-bpffw\") pod \"payload-processing-e2e-mt-2a090f42-5d466c5697-q46lj\" (UID: \"06632265-dbd0-42f1-a9b5-7033ae63bcee\") " pod="openshift-ingress/payload-processing-e2e-mt-2a090f42-5d466c5697-q46lj" Sep 09 10:48:56.573706 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:56.573590 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a090f42-c644cb48b-8np6k" Sep 09 10:48:56.601069 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:56.601035 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2a090f42-5d466c5697-q46lj" Sep 09 10:48:56.852345 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:56.852320 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2a090f42-c644cb48b-8np6k"] Sep 09 10:48:56.855313 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:48:56.855279 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1866cfd9_c3e2_469a_baad_42bd6efce9c6.slice/crio-8c3dac6fb5b1b54391bdadd778c0917000bdb651377ce4372ea141567663310a WatchSource:0}: Error finding container 8c3dac6fb5b1b54391bdadd778c0917000bdb651377ce4372ea141567663310a: Status 404 returned error can't find the container with id 8c3dac6fb5b1b54391bdadd778c0917000bdb651377ce4372ea141567663310a Sep 09 10:48:56.891089 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:56.891060 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2a090f42-5d466c5697-q46lj"] Sep 09 10:48:56.893288 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:48:56.893260 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06632265_dbd0_42f1_a9b5_7033ae63bcee.slice/crio-147e716e3825a524955abddc4582e8d3e60cc78b4d7ab92d09aa22f45c9ac546 WatchSource:0}: Error finding container 147e716e3825a524955abddc4582e8d3e60cc78b4d7ab92d09aa22f45c9ac546: Status 404 returned error can't find the container with id 147e716e3825a524955abddc4582e8d3e60cc78b4d7ab92d09aa22f45c9ac546 Sep 09 10:48:57.217163 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:57.217058 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2a090f42-5d466c5697-q46lj" event={"ID":"06632265-dbd0-42f1-a9b5-7033ae63bcee","Type":"ContainerStarted","Data":"646de26587a7bd996a451f0317bc4ccd41762a6f70c95a73e20eb9a736a1a8ad"} Sep 09 10:48:57.217319 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:57.217246 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-2a090f42-5d466c5697-q46lj" Sep 09 10:48:57.217319 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:57.217283 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2a090f42-5d466c5697-q46lj" event={"ID":"06632265-dbd0-42f1-a9b5-7033ae63bcee","Type":"ContainerStarted","Data":"147e716e3825a524955abddc4582e8d3e60cc78b4d7ab92d09aa22f45c9ac546"} Sep 09 10:48:57.218693 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:57.218663 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a090f42-c644cb48b-8np6k" event={"ID":"1866cfd9-c3e2-469a-baad-42bd6efce9c6","Type":"ContainerStarted","Data":"f31bbb41a143f2cdfd6f2355e5b4c9f27362c870a6ff1a9694cdc63c2318906e"} Sep 09 10:48:57.218807 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:57.218715 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a090f42-c644cb48b-8np6k" Sep 09 10:48:57.218807 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:57.218732 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a090f42-c644cb48b-8np6k" event={"ID":"1866cfd9-c3e2-469a-baad-42bd6efce9c6","Type":"ContainerStarted","Data":"8c3dac6fb5b1b54391bdadd778c0917000bdb651377ce4372ea141567663310a"} Sep 09 10:48:57.237002 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:57.236947 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-2a090f42-5d466c5697-q46lj" podStartSLOduration=1.236930382 podStartE2EDuration="1.236930382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:48:57.234972474 +0000 UTC m=+1377.409574799" watchObservedRunningTime="2026-09-09 10:48:57.236930382 +0000 UTC m=+1377.411532707" Sep 09 10:48:57.254771 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:48:57.254699 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a090f42-c644cb48b-8np6k" podStartSLOduration=1.254679916 podStartE2EDuration="1.254679916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:48:57.252951903 +0000 UTC m=+1377.427554229" watchObservedRunningTime="2026-09-09 10:48:57.254679916 +0000 UTC m=+1377.429282240" Sep 09 10:49:06.211551 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:06.211491 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1eb21d-846bcbb58-p64nc" Sep 09 10:49:06.212119 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:06.211637 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1eb21d-57fcdc7b5-n5fr8" Sep 09 10:49:08.226545 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:08.226490 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a090f42-c644cb48b-8np6k" Sep 09 10:49:08.226916 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:08.226593 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-2a090f42-5d466c5697-q46lj" Sep 09 10:49:22.766168 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:22.766129 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-535b75-78f95fcfb6-5lpzd"] Sep 09 10:49:22.771192 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:22.771171 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-535b75-78f95fcfb6-5lpzd" Sep 09 10:49:22.773786 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:22.773762 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-535b75\"" Sep 09 10:49:22.774162 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:22.774141 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-535b75-dockercfg-8kcfb\"" Sep 09 10:49:22.778787 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:22.778755 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-535b75-78f95fcfb6-5lpzd"] Sep 09 10:49:22.793383 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:22.793351 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-535b75-5df86cb7f9-npkvs"] Sep 09 10:49:22.796924 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:22.796902 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-535b75-5df86cb7f9-npkvs" Sep 09 10:49:22.807763 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:22.807735 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-535b75-5df86cb7f9-npkvs"] Sep 09 10:49:22.821438 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:22.821400 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/045ab401-6d45-4953-a1f0-7e590bf20c24-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-535b75-78f95fcfb6-5lpzd\" (UID: \"045ab401-6d45-4953-a1f0-7e590bf20c24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-535b75-78f95fcfb6-5lpzd" Sep 09 10:49:22.821653 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:22.821453 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6vds\" (UniqueName: \"kubernetes.io/projected/045ab401-6d45-4953-a1f0-7e590bf20c24-kube-api-access-h6vds\") pod \"payload-pre-processing-e2e-shared-b-w3-535b75-78f95fcfb6-5lpzd\" (UID: \"045ab401-6d45-4953-a1f0-7e590bf20c24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-535b75-78f95fcfb6-5lpzd" Sep 09 10:49:22.922364 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:22.922326 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5460687d-4874-49df-8f22-38e79231b6a4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-535b75-5df86cb7f9-npkvs\" (UID: \"5460687d-4874-49df-8f22-38e79231b6a4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-535b75-5df86cb7f9-npkvs" Sep 09 10:49:22.922364 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:22.922372 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/045ab401-6d45-4953-a1f0-7e590bf20c24-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-535b75-78f95fcfb6-5lpzd\" (UID: \"045ab401-6d45-4953-a1f0-7e590bf20c24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-535b75-78f95fcfb6-5lpzd" Sep 09 10:49:22.922689 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:22.922395 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp52h\" (UniqueName: \"kubernetes.io/projected/5460687d-4874-49df-8f22-38e79231b6a4-kube-api-access-wp52h\") pod \"payload-processing-e2e-shared-b-w3-535b75-5df86cb7f9-npkvs\" (UID: \"5460687d-4874-49df-8f22-38e79231b6a4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-535b75-5df86cb7f9-npkvs" Sep 09 10:49:22.922689 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:22.922417 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6vds\" (UniqueName: \"kubernetes.io/projected/045ab401-6d45-4953-a1f0-7e590bf20c24-kube-api-access-h6vds\") pod \"payload-pre-processing-e2e-shared-b-w3-535b75-78f95fcfb6-5lpzd\" (UID: \"045ab401-6d45-4953-a1f0-7e590bf20c24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-535b75-78f95fcfb6-5lpzd" Sep 09 10:49:22.923179 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:22.923153 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/045ab401-6d45-4953-a1f0-7e590bf20c24-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-535b75-78f95fcfb6-5lpzd\" (UID: \"045ab401-6d45-4953-a1f0-7e590bf20c24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-535b75-78f95fcfb6-5lpzd" Sep 09 10:49:22.930768 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:22.930744 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6vds\" (UniqueName: \"kubernetes.io/projected/045ab401-6d45-4953-a1f0-7e590bf20c24-kube-api-access-h6vds\") pod \"payload-pre-processing-e2e-shared-b-w3-535b75-78f95fcfb6-5lpzd\" (UID: \"045ab401-6d45-4953-a1f0-7e590bf20c24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-535b75-78f95fcfb6-5lpzd" Sep 09 10:49:23.023381 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:23.023278 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5460687d-4874-49df-8f22-38e79231b6a4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-535b75-5df86cb7f9-npkvs\" (UID: \"5460687d-4874-49df-8f22-38e79231b6a4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-535b75-5df86cb7f9-npkvs" Sep 09 10:49:23.023381 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:23.023330 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp52h\" (UniqueName: \"kubernetes.io/projected/5460687d-4874-49df-8f22-38e79231b6a4-kube-api-access-wp52h\") pod \"payload-processing-e2e-shared-b-w3-535b75-5df86cb7f9-npkvs\" (UID: \"5460687d-4874-49df-8f22-38e79231b6a4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-535b75-5df86cb7f9-npkvs" Sep 09 10:49:23.024071 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:23.024045 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5460687d-4874-49df-8f22-38e79231b6a4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-535b75-5df86cb7f9-npkvs\" (UID: \"5460687d-4874-49df-8f22-38e79231b6a4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-535b75-5df86cb7f9-npkvs" Sep 09 10:49:23.032036 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:23.032013 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp52h\" (UniqueName: \"kubernetes.io/projected/5460687d-4874-49df-8f22-38e79231b6a4-kube-api-access-wp52h\") pod \"payload-processing-e2e-shared-b-w3-535b75-5df86cb7f9-npkvs\" (UID: \"5460687d-4874-49df-8f22-38e79231b6a4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-535b75-5df86cb7f9-npkvs" Sep 09 10:49:23.083313 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:23.083279 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-535b75-78f95fcfb6-5lpzd" Sep 09 10:49:23.107658 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:23.107624 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-535b75-5df86cb7f9-npkvs" Sep 09 10:49:23.356882 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:23.356861 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-535b75-78f95fcfb6-5lpzd"] Sep 09 10:49:23.358860 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:49:23.358831 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod045ab401_6d45_4953_a1f0_7e590bf20c24.slice/crio-0f931d6f4a5534f504b4695c15275bf12ce3375c5d273cd311282612fd165c90 WatchSource:0}: Error finding container 0f931d6f4a5534f504b4695c15275bf12ce3375c5d273cd311282612fd165c90: Status 404 returned error can't find the container with id 0f931d6f4a5534f504b4695c15275bf12ce3375c5d273cd311282612fd165c90 Sep 09 10:49:23.390943 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:23.390917 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-535b75-5df86cb7f9-npkvs"] Sep 09 10:49:23.393618 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:49:23.393585 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5460687d_4874_49df_8f22_38e79231b6a4.slice/crio-b839e761d4dafe9e5fd84eebc61abc793913f6b295e90e4672f56654f87b6c1c WatchSource:0}: Error finding container b839e761d4dafe9e5fd84eebc61abc793913f6b295e90e4672f56654f87b6c1c: Status 404 returned error can't find the container with id b839e761d4dafe9e5fd84eebc61abc793913f6b295e90e4672f56654f87b6c1c Sep 09 10:49:24.368456 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:24.368416 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-535b75-78f95fcfb6-5lpzd" event={"ID":"045ab401-6d45-4953-a1f0-7e590bf20c24","Type":"ContainerStarted","Data":"90f32e03a31c89046a50545d8c025a994d1bc3f368e67b455a5c1162378c5681"} Sep 09 10:49:24.368963 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:24.368481 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-535b75-78f95fcfb6-5lpzd" event={"ID":"045ab401-6d45-4953-a1f0-7e590bf20c24","Type":"ContainerStarted","Data":"0f931d6f4a5534f504b4695c15275bf12ce3375c5d273cd311282612fd165c90"} Sep 09 10:49:24.368963 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:24.368564 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-535b75-78f95fcfb6-5lpzd" Sep 09 10:49:24.370201 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:24.370164 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-535b75-5df86cb7f9-npkvs" event={"ID":"5460687d-4874-49df-8f22-38e79231b6a4","Type":"ContainerStarted","Data":"bfbd32ed5484e5dcd89253bcda4e15a7ae21d2cd0ee00a698d1ba22426c75593"} Sep 09 10:49:24.370361 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:24.370226 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-535b75-5df86cb7f9-npkvs" event={"ID":"5460687d-4874-49df-8f22-38e79231b6a4","Type":"ContainerStarted","Data":"b839e761d4dafe9e5fd84eebc61abc793913f6b295e90e4672f56654f87b6c1c"} Sep 09 10:49:24.370361 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:24.370257 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-535b75-5df86cb7f9-npkvs" Sep 09 10:49:24.388622 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:24.388573 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-535b75-78f95fcfb6-5lpzd" podStartSLOduration=2.388552285 podStartE2EDuration="2.388552285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:49:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:49:24.386871197 +0000 UTC m=+1404.561473522" watchObservedRunningTime="2026-09-09 10:49:24.388552285 +0000 UTC m=+1404.563154663" Sep 09 10:49:24.407530 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:24.407458 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-535b75-5df86cb7f9-npkvs" podStartSLOduration=2.407442275 podStartE2EDuration="2.407442275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:49:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:49:24.405736914 +0000 UTC m=+1404.580339269" watchObservedRunningTime="2026-09-09 10:49:24.407442275 +0000 UTC m=+1404.582044600" Sep 09 10:49:26.783578 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:26.783546 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a090f42-6768p"] Sep 09 10:49:26.787075 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:26.787057 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a090f42-6768p" Sep 09 10:49:26.789434 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:26.789407 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kq7qv\"" Sep 09 10:49:26.789610 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:26.789523 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 10:49:26.794295 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:26.794269 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a090f42-6768p"] Sep 09 10:49:26.857645 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:26.857602 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bc7e10c-6ac9-4b3d-b493-c1b6d1f71bf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2a090f42-6768p\" (UID: \"6bc7e10c-6ac9-4b3d-b493-c1b6d1f71bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a090f42-6768p" Sep 09 10:49:26.958891 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:26.958861 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bc7e10c-6ac9-4b3d-b493-c1b6d1f71bf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2a090f42-6768p\" (UID: \"6bc7e10c-6ac9-4b3d-b493-c1b6d1f71bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a090f42-6768p" Sep 09 10:49:26.959490 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:26.959470 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bc7e10c-6ac9-4b3d-b493-c1b6d1f71bf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2a090f42-6768p\" (UID: \"6bc7e10c-6ac9-4b3d-b493-c1b6d1f71bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a090f42-6768p" Sep 09 10:49:27.099610 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:27.099575 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a090f42-6768p" Sep 09 10:49:27.370946 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:27.370859 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a090f42-6768p"] Sep 09 10:49:27.374626 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:49:27.374600 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc7e10c_6ac9_4b3d_b493_c1b6d1f71bf0.slice/crio-717b86d736374a1f04789c5eaa90bc699910ee946183f6aead14579b37f11e68 WatchSource:0}: Error finding container 717b86d736374a1f04789c5eaa90bc699910ee946183f6aead14579b37f11e68: Status 404 returned error can't find the container with id 717b86d736374a1f04789c5eaa90bc699910ee946183f6aead14579b37f11e68 Sep 09 10:49:27.388562 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:27.388528 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a090f42-6768p" event={"ID":"6bc7e10c-6ac9-4b3d-b493-c1b6d1f71bf0","Type":"ContainerStarted","Data":"717b86d736374a1f04789c5eaa90bc699910ee946183f6aead14579b37f11e68"} Sep 09 10:49:28.395711 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:28.395668 2559 generic.go:356] "Generic (PLEG): container finished" podID="6bc7e10c-6ac9-4b3d-b493-c1b6d1f71bf0" containerID="e2e548f8fdc85d6bd5b10c78470dac61eab5b0e96747ce8b09a1816c766d206f" exitCode=0 Sep 09 10:49:28.396108 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:28.395734 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a090f42-6768p" event={"ID":"6bc7e10c-6ac9-4b3d-b493-c1b6d1f71bf0","Type":"ContainerDied","Data":"e2e548f8fdc85d6bd5b10c78470dac61eab5b0e96747ce8b09a1816c766d206f"} Sep 09 10:49:29.531100 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:29.531077 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a090f42-6768p" Sep 09 10:49:29.683080 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:29.682995 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bc7e10c-6ac9-4b3d-b493-c1b6d1f71bf0-maas-api-service-ca\") pod \"6bc7e10c-6ac9-4b3d-b493-c1b6d1f71bf0\" (UID: \"6bc7e10c-6ac9-4b3d-b493-c1b6d1f71bf0\") " Sep 09 10:49:29.683368 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:29.683343 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bc7e10c-6ac9-4b3d-b493-c1b6d1f71bf0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6bc7e10c-6ac9-4b3d-b493-c1b6d1f71bf0" (UID: "6bc7e10c-6ac9-4b3d-b493-c1b6d1f71bf0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:49:29.784060 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:29.784023 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bc7e10c-6ac9-4b3d-b493-c1b6d1f71bf0-maas-api-service-ca\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:49:30.407087 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:30.407059 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a090f42-6768p" Sep 09 10:49:30.407087 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:30.407083 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a090f42-6768p" event={"ID":"6bc7e10c-6ac9-4b3d-b493-c1b6d1f71bf0","Type":"ContainerDied","Data":"717b86d736374a1f04789c5eaa90bc699910ee946183f6aead14579b37f11e68"} Sep 09 10:49:30.407288 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:30.407131 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="717b86d736374a1f04789c5eaa90bc699910ee946183f6aead14579b37f11e68" Sep 09 10:49:35.378649 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:35.378614 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-535b75-5df86cb7f9-npkvs" Sep 09 10:49:35.379019 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:35.378688 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-535b75-78f95fcfb6-5lpzd" Sep 09 10:49:37.425227 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:37.425191 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2a090f42-5d466c5697-q46lj"] Sep 09 10:49:37.425704 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:37.425533 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-2a090f42-5d466c5697-q46lj" podUID="06632265-dbd0-42f1-a9b5-7033ae63bcee" containerName="payload-processing" containerID="cri-o://646de26587a7bd996a451f0317bc4ccd41762a6f70c95a73e20eb9a736a1a8ad" gracePeriod=30 Sep 09 10:49:37.432947 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:37.432879 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2a090f42-c644cb48b-8np6k"] Sep 09 10:49:37.433103 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:37.433077 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a090f42-c644cb48b-8np6k" podUID="1866cfd9-c3e2-469a-baad-42bd6efce9c6" containerName="payload-pre-processing" containerID="cri-o://f31bbb41a143f2cdfd6f2355e5b4c9f27362c870a6ff1a9694cdc63c2318906e" gracePeriod=30 Sep 09 10:49:37.648445 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:37.648410 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a090f42-6768p"] Sep 09 10:49:37.656711 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:37.656678 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a090f42-6768p"] Sep 09 10:49:37.704748 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:37.704724 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2a090f42-5d466c5697-q46lj" Sep 09 10:49:37.707693 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:37.707674 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a090f42-c644cb48b-8np6k" Sep 09 10:49:37.855902 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:37.855865 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrslq\" (UniqueName: \"kubernetes.io/projected/1866cfd9-c3e2-469a-baad-42bd6efce9c6-kube-api-access-qrslq\") pod \"1866cfd9-c3e2-469a-baad-42bd6efce9c6\" (UID: \"1866cfd9-c3e2-469a-baad-42bd6efce9c6\") " Sep 09 10:49:37.856071 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:37.855965 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1866cfd9-c3e2-469a-baad-42bd6efce9c6-plugins-config-volume\") pod \"1866cfd9-c3e2-469a-baad-42bd6efce9c6\" (UID: \"1866cfd9-c3e2-469a-baad-42bd6efce9c6\") " Sep 09 10:49:37.856071 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:37.855996 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06632265-dbd0-42f1-a9b5-7033ae63bcee-plugins-config-volume\") pod \"06632265-dbd0-42f1-a9b5-7033ae63bcee\" (UID: \"06632265-dbd0-42f1-a9b5-7033ae63bcee\") " Sep 09 10:49:37.856071 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:37.856030 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpffw\" (UniqueName: \"kubernetes.io/projected/06632265-dbd0-42f1-a9b5-7033ae63bcee-kube-api-access-bpffw\") pod \"06632265-dbd0-42f1-a9b5-7033ae63bcee\" (UID: \"06632265-dbd0-42f1-a9b5-7033ae63bcee\") " Sep 09 10:49:37.856394 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:37.856355 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1866cfd9-c3e2-469a-baad-42bd6efce9c6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1866cfd9-c3e2-469a-baad-42bd6efce9c6" (UID: "1866cfd9-c3e2-469a-baad-42bd6efce9c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:49:37.856394 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:37.856358 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06632265-dbd0-42f1-a9b5-7033ae63bcee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "06632265-dbd0-42f1-a9b5-7033ae63bcee" (UID: "06632265-dbd0-42f1-a9b5-7033ae63bcee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:49:37.858081 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:37.858062 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1866cfd9-c3e2-469a-baad-42bd6efce9c6-kube-api-access-qrslq" (OuterVolumeSpecName: "kube-api-access-qrslq") pod "1866cfd9-c3e2-469a-baad-42bd6efce9c6" (UID: "1866cfd9-c3e2-469a-baad-42bd6efce9c6"). InnerVolumeSpecName "kube-api-access-qrslq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:49:37.858160 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:37.858120 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06632265-dbd0-42f1-a9b5-7033ae63bcee-kube-api-access-bpffw" (OuterVolumeSpecName: "kube-api-access-bpffw") pod "06632265-dbd0-42f1-a9b5-7033ae63bcee" (UID: "06632265-dbd0-42f1-a9b5-7033ae63bcee"). InnerVolumeSpecName "kube-api-access-bpffw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:49:37.956826 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:37.956738 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1866cfd9-c3e2-469a-baad-42bd6efce9c6-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:49:37.956826 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:37.956779 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06632265-dbd0-42f1-a9b5-7033ae63bcee-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:49:37.956826 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:37.956795 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpffw\" (UniqueName: \"kubernetes.io/projected/06632265-dbd0-42f1-a9b5-7033ae63bcee-kube-api-access-bpffw\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:49:37.956826 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:37.956809 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrslq\" (UniqueName: \"kubernetes.io/projected/1866cfd9-c3e2-469a-baad-42bd6efce9c6-kube-api-access-qrslq\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:49:38.334246 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:38.334214 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bc7e10c-6ac9-4b3d-b493-c1b6d1f71bf0" path="/var/lib/kubelet/pods/6bc7e10c-6ac9-4b3d-b493-c1b6d1f71bf0/volumes" Sep 09 10:49:38.460388 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:38.460351 2559 generic.go:356] "Generic (PLEG): container finished" podID="06632265-dbd0-42f1-a9b5-7033ae63bcee" containerID="646de26587a7bd996a451f0317bc4ccd41762a6f70c95a73e20eb9a736a1a8ad" exitCode=0 Sep 09 10:49:38.460867 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:38.460398 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2a090f42-5d466c5697-q46lj" Sep 09 10:49:38.460867 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:38.460434 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2a090f42-5d466c5697-q46lj" event={"ID":"06632265-dbd0-42f1-a9b5-7033ae63bcee","Type":"ContainerDied","Data":"646de26587a7bd996a451f0317bc4ccd41762a6f70c95a73e20eb9a736a1a8ad"} Sep 09 10:49:38.460867 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:38.460483 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2a090f42-5d466c5697-q46lj" event={"ID":"06632265-dbd0-42f1-a9b5-7033ae63bcee","Type":"ContainerDied","Data":"147e716e3825a524955abddc4582e8d3e60cc78b4d7ab92d09aa22f45c9ac546"} Sep 09 10:49:38.460867 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:38.460519 2559 scope.go:117] "RemoveContainer" containerID="646de26587a7bd996a451f0317bc4ccd41762a6f70c95a73e20eb9a736a1a8ad" Sep 09 10:49:38.462121 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:38.462097 2559 generic.go:356] "Generic (PLEG): container finished" podID="1866cfd9-c3e2-469a-baad-42bd6efce9c6" containerID="f31bbb41a143f2cdfd6f2355e5b4c9f27362c870a6ff1a9694cdc63c2318906e" exitCode=0 Sep 09 10:49:38.462241 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:38.462129 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a090f42-c644cb48b-8np6k" event={"ID":"1866cfd9-c3e2-469a-baad-42bd6efce9c6","Type":"ContainerDied","Data":"f31bbb41a143f2cdfd6f2355e5b4c9f27362c870a6ff1a9694cdc63c2318906e"} Sep 09 10:49:38.462241 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:38.462140 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a090f42-c644cb48b-8np6k" Sep 09 10:49:38.462241 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:38.462184 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a090f42-c644cb48b-8np6k" event={"ID":"1866cfd9-c3e2-469a-baad-42bd6efce9c6","Type":"ContainerDied","Data":"8c3dac6fb5b1b54391bdadd778c0917000bdb651377ce4372ea141567663310a"} Sep 09 10:49:38.470031 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:38.470013 2559 scope.go:117] "RemoveContainer" containerID="646de26587a7bd996a451f0317bc4ccd41762a6f70c95a73e20eb9a736a1a8ad" Sep 09 10:49:38.470288 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:49:38.470268 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"646de26587a7bd996a451f0317bc4ccd41762a6f70c95a73e20eb9a736a1a8ad\": container with ID starting with 646de26587a7bd996a451f0317bc4ccd41762a6f70c95a73e20eb9a736a1a8ad not found: ID does not exist" containerID="646de26587a7bd996a451f0317bc4ccd41762a6f70c95a73e20eb9a736a1a8ad" Sep 09 10:49:38.470369 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:38.470300 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"646de26587a7bd996a451f0317bc4ccd41762a6f70c95a73e20eb9a736a1a8ad"} err="failed to get container status \"646de26587a7bd996a451f0317bc4ccd41762a6f70c95a73e20eb9a736a1a8ad\": rpc error: code = NotFound desc = could not find container \"646de26587a7bd996a451f0317bc4ccd41762a6f70c95a73e20eb9a736a1a8ad\": container with ID starting with 646de26587a7bd996a451f0317bc4ccd41762a6f70c95a73e20eb9a736a1a8ad not found: ID does not exist" Sep 09 10:49:38.470369 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:38.470323 2559 scope.go:117] "RemoveContainer" containerID="f31bbb41a143f2cdfd6f2355e5b4c9f27362c870a6ff1a9694cdc63c2318906e" Sep 09 10:49:38.484040 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:38.484024 2559 scope.go:117] "RemoveContainer" containerID="f31bbb41a143f2cdfd6f2355e5b4c9f27362c870a6ff1a9694cdc63c2318906e" Sep 09 10:49:38.484310 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:49:38.484292 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f31bbb41a143f2cdfd6f2355e5b4c9f27362c870a6ff1a9694cdc63c2318906e\": container with ID starting with f31bbb41a143f2cdfd6f2355e5b4c9f27362c870a6ff1a9694cdc63c2318906e not found: ID does not exist" containerID="f31bbb41a143f2cdfd6f2355e5b4c9f27362c870a6ff1a9694cdc63c2318906e" Sep 09 10:49:38.484378 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:38.484315 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f31bbb41a143f2cdfd6f2355e5b4c9f27362c870a6ff1a9694cdc63c2318906e"} err="failed to get container status \"f31bbb41a143f2cdfd6f2355e5b4c9f27362c870a6ff1a9694cdc63c2318906e\": rpc error: code = NotFound desc = could not find container \"f31bbb41a143f2cdfd6f2355e5b4c9f27362c870a6ff1a9694cdc63c2318906e\": container with ID starting with f31bbb41a143f2cdfd6f2355e5b4c9f27362c870a6ff1a9694cdc63c2318906e not found: ID does not exist" Sep 09 10:49:38.487843 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:38.487822 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2a090f42-c644cb48b-8np6k"] Sep 09 10:49:38.490240 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:38.490216 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2a090f42-c644cb48b-8np6k"] Sep 09 10:49:38.501339 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:38.501315 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2a090f42-5d466c5697-q46lj"] Sep 09 10:49:38.503772 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:38.503753 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2a090f42-5d466c5697-q46lj"] Sep 09 10:49:40.335487 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:40.335454 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06632265-dbd0-42f1-a9b5-7033ae63bcee" path="/var/lib/kubelet/pods/06632265-dbd0-42f1-a9b5-7033ae63bcee/volumes" Sep 09 10:49:40.335863 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:40.335820 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1866cfd9-c3e2-469a-baad-42bd6efce9c6" path="/var/lib/kubelet/pods/1866cfd9-c3e2-469a-baad-42bd6efce9c6/volumes" Sep 09 10:49:50.638234 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.638198 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv"] Sep 09 10:49:50.638829 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.638627 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06632265-dbd0-42f1-a9b5-7033ae63bcee" containerName="payload-processing" Sep 09 10:49:50.638829 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.638642 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="06632265-dbd0-42f1-a9b5-7033ae63bcee" containerName="payload-processing" Sep 09 10:49:50.638829 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.638654 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bc7e10c-6ac9-4b3d-b493-c1b6d1f71bf0" containerName="revoke-keys" Sep 09 10:49:50.638829 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.638659 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bc7e10c-6ac9-4b3d-b493-c1b6d1f71bf0" containerName="revoke-keys" Sep 09 10:49:50.638829 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.638670 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1866cfd9-c3e2-469a-baad-42bd6efce9c6" containerName="payload-pre-processing" Sep 09 10:49:50.638829 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.638676 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1866cfd9-c3e2-469a-baad-42bd6efce9c6" containerName="payload-pre-processing" Sep 09 10:49:50.638829 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.638736 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1866cfd9-c3e2-469a-baad-42bd6efce9c6" containerName="payload-pre-processing" Sep 09 10:49:50.638829 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.638746 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bc7e10c-6ac9-4b3d-b493-c1b6d1f71bf0" containerName="revoke-keys" Sep 09 10:49:50.638829 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.638760 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="06632265-dbd0-42f1-a9b5-7033ae63bcee" containerName="payload-processing" Sep 09 10:49:50.643154 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.643131 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" Sep 09 10:49:50.646994 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.646964 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1eb21d\"/\"kube-root-ca.crt\"" Sep 09 10:49:50.647143 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.647028 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1eb21d\"/\"default-dockercfg-kmqqb\"" Sep 09 10:49:50.647143 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.647043 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1eb21d\"/\"openshift-service-ca.crt\"" Sep 09 10:49:50.647143 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.647043 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1eb21d\"/\"auth-test-model-1eb21d-kserve-self-signed-certs\"" Sep 09 10:49:50.655569 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.655542 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv"] Sep 09 10:49:50.770269 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.770232 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f0820db-9a67-4291-ac82-d4040621ed03-home\") pod \"auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv\" (UID: \"9f0820db-9a67-4291-ac82-d4040621ed03\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" Sep 09 10:49:50.770449 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.770284 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f0820db-9a67-4291-ac82-d4040621ed03-model-cache\") pod \"auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv\" (UID: \"9f0820db-9a67-4291-ac82-d4040621ed03\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" Sep 09 10:49:50.770449 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.770341 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f0820db-9a67-4291-ac82-d4040621ed03-dshm\") pod \"auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv\" (UID: \"9f0820db-9a67-4291-ac82-d4040621ed03\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" Sep 09 10:49:50.770449 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.770393 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f0820db-9a67-4291-ac82-d4040621ed03-tls-certs\") pod \"auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv\" (UID: \"9f0820db-9a67-4291-ac82-d4040621ed03\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" Sep 09 10:49:50.770449 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.770419 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms7lp\" (UniqueName: \"kubernetes.io/projected/9f0820db-9a67-4291-ac82-d4040621ed03-kube-api-access-ms7lp\") pod \"auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv\" (UID: \"9f0820db-9a67-4291-ac82-d4040621ed03\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" Sep 09 10:49:50.770449 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.770438 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f0820db-9a67-4291-ac82-d4040621ed03-tmp-dir\") pod \"auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv\" (UID: \"9f0820db-9a67-4291-ac82-d4040621ed03\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" Sep 09 10:49:50.871323 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.871277 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f0820db-9a67-4291-ac82-d4040621ed03-home\") pod \"auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv\" (UID: \"9f0820db-9a67-4291-ac82-d4040621ed03\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" Sep 09 10:49:50.871556 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.871340 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f0820db-9a67-4291-ac82-d4040621ed03-model-cache\") pod \"auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv\" (UID: \"9f0820db-9a67-4291-ac82-d4040621ed03\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" Sep 09 10:49:50.871556 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.871375 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f0820db-9a67-4291-ac82-d4040621ed03-dshm\") pod \"auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv\" (UID: \"9f0820db-9a67-4291-ac82-d4040621ed03\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" Sep 09 10:49:50.871556 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.871416 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f0820db-9a67-4291-ac82-d4040621ed03-tls-certs\") pod \"auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv\" (UID: \"9f0820db-9a67-4291-ac82-d4040621ed03\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" Sep 09 10:49:50.871556 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.871444 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms7lp\" (UniqueName: \"kubernetes.io/projected/9f0820db-9a67-4291-ac82-d4040621ed03-kube-api-access-ms7lp\") pod \"auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv\" (UID: \"9f0820db-9a67-4291-ac82-d4040621ed03\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" Sep 09 10:49:50.871556 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.871469 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f0820db-9a67-4291-ac82-d4040621ed03-tmp-dir\") pod \"auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv\" (UID: \"9f0820db-9a67-4291-ac82-d4040621ed03\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" Sep 09 10:49:50.871863 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.871776 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f0820db-9a67-4291-ac82-d4040621ed03-home\") pod \"auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv\" (UID: \"9f0820db-9a67-4291-ac82-d4040621ed03\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" Sep 09 10:49:50.871863 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.871853 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f0820db-9a67-4291-ac82-d4040621ed03-tmp-dir\") pod \"auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv\" (UID: \"9f0820db-9a67-4291-ac82-d4040621ed03\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" Sep 09 10:49:50.871958 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.871914 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f0820db-9a67-4291-ac82-d4040621ed03-model-cache\") pod \"auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv\" (UID: \"9f0820db-9a67-4291-ac82-d4040621ed03\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" Sep 09 10:49:50.873594 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.873573 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f0820db-9a67-4291-ac82-d4040621ed03-dshm\") pod \"auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv\" (UID: \"9f0820db-9a67-4291-ac82-d4040621ed03\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" Sep 09 10:49:50.873880 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.873864 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f0820db-9a67-4291-ac82-d4040621ed03-tls-certs\") pod \"auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv\" (UID: \"9f0820db-9a67-4291-ac82-d4040621ed03\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" Sep 09 10:49:50.880397 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.880364 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms7lp\" (UniqueName: \"kubernetes.io/projected/9f0820db-9a67-4291-ac82-d4040621ed03-kube-api-access-ms7lp\") pod \"auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv\" (UID: \"9f0820db-9a67-4291-ac82-d4040621ed03\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" Sep 09 10:49:50.955851 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:50.955762 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" Sep 09 10:49:51.232463 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:51.229230 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv"] Sep 09 10:49:51.532869 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:51.532768 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" event={"ID":"9f0820db-9a67-4291-ac82-d4040621ed03","Type":"ContainerStarted","Data":"0f33bc0327720d72e80c7e2d3bbb0e1b27239fdc7df77c7e0bf877038385e7a2"} Sep 09 10:49:51.532869 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:51.532822 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" event={"ID":"9f0820db-9a67-4291-ac82-d4040621ed03","Type":"ContainerStarted","Data":"387237739d5cf34152f11f53ea8b342ad7c812f92e455bce4851cd9385590e20"} Sep 09 10:49:51.552873 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:49:51.552821 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" podStartSLOduration=1.5528020759999999 podStartE2EDuration="1.552802076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:49:51.551288164 +0000 UTC m=+1431.725890491" watchObservedRunningTime="2026-09-09 10:49:51.552802076 +0000 UTC m=+1431.727404400" Sep 09 10:50:00.956805 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:00.956766 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" Sep 09 10:50:00.956805 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:00.956816 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" Sep 09 10:50:00.969351 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:00.969324 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" Sep 09 10:50:01.597952 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:01.597919 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" Sep 09 10:50:04.661030 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:04.660991 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-62f893-6757df5f6b-vt8nx"] Sep 09 10:50:04.665887 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:04.665856 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-62f893-6757df5f6b-vt8nx" Sep 09 10:50:04.668595 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:04.668572 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-62f893\"" Sep 09 10:50:04.668810 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:04.668757 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-62f893-dockercfg-mj6tg\"" Sep 09 10:50:04.673855 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:04.673827 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tbs9\" (UniqueName: \"kubernetes.io/projected/5d14c4c8-3a61-48b6-a94d-55b5edd08b46-kube-api-access-5tbs9\") pod \"payload-pre-processing-e2e-shared-a-w2-62f893-6757df5f6b-vt8nx\" (UID: \"5d14c4c8-3a61-48b6-a94d-55b5edd08b46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-62f893-6757df5f6b-vt8nx" Sep 09 10:50:04.673992 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:04.673969 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d14c4c8-3a61-48b6-a94d-55b5edd08b46-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-62f893-6757df5f6b-vt8nx\" (UID: \"5d14c4c8-3a61-48b6-a94d-55b5edd08b46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-62f893-6757df5f6b-vt8nx" Sep 09 10:50:04.676128 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:04.676100 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-62f893-6757df5f6b-vt8nx"] Sep 09 10:50:04.775074 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:04.775037 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d14c4c8-3a61-48b6-a94d-55b5edd08b46-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-62f893-6757df5f6b-vt8nx\" (UID: \"5d14c4c8-3a61-48b6-a94d-55b5edd08b46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-62f893-6757df5f6b-vt8nx" Sep 09 10:50:04.775260 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:04.775100 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tbs9\" (UniqueName: \"kubernetes.io/projected/5d14c4c8-3a61-48b6-a94d-55b5edd08b46-kube-api-access-5tbs9\") pod \"payload-pre-processing-e2e-shared-a-w2-62f893-6757df5f6b-vt8nx\" (UID: \"5d14c4c8-3a61-48b6-a94d-55b5edd08b46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-62f893-6757df5f6b-vt8nx" Sep 09 10:50:04.775827 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:04.775805 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d14c4c8-3a61-48b6-a94d-55b5edd08b46-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-62f893-6757df5f6b-vt8nx\" (UID: \"5d14c4c8-3a61-48b6-a94d-55b5edd08b46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-62f893-6757df5f6b-vt8nx" Sep 09 10:50:04.784383 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:04.784343 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tbs9\" (UniqueName: \"kubernetes.io/projected/5d14c4c8-3a61-48b6-a94d-55b5edd08b46-kube-api-access-5tbs9\") pod \"payload-pre-processing-e2e-shared-a-w2-62f893-6757df5f6b-vt8nx\" (UID: \"5d14c4c8-3a61-48b6-a94d-55b5edd08b46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-62f893-6757df5f6b-vt8nx" Sep 09 10:50:04.977549 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:04.977439 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-62f893-6757df5f6b-vt8nx" Sep 09 10:50:05.259118 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:05.259086 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-62f893-6757df5f6b-vt8nx"] Sep 09 10:50:05.261821 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:50:05.261794 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d14c4c8_3a61_48b6_a94d_55b5edd08b46.slice/crio-507922e20450a574f155cedcbad248064aba2be1e31b05d0be485ca088dddb44 WatchSource:0}: Error finding container 507922e20450a574f155cedcbad248064aba2be1e31b05d0be485ca088dddb44: Status 404 returned error can't find the container with id 507922e20450a574f155cedcbad248064aba2be1e31b05d0be485ca088dddb44 Sep 09 10:50:05.615318 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:05.615280 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-62f893-6757df5f6b-vt8nx" event={"ID":"5d14c4c8-3a61-48b6-a94d-55b5edd08b46","Type":"ContainerStarted","Data":"f254ac4b72db040f828a087b14ce6d6c79ef3769640f05eb9bcb4cd2faa4a33b"} Sep 09 10:50:05.615834 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:05.615332 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-62f893-6757df5f6b-vt8nx" event={"ID":"5d14c4c8-3a61-48b6-a94d-55b5edd08b46","Type":"ContainerStarted","Data":"507922e20450a574f155cedcbad248064aba2be1e31b05d0be485ca088dddb44"} Sep 09 10:50:05.615834 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:05.615394 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-62f893-6757df5f6b-vt8nx" Sep 09 10:50:05.633006 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:05.632949 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-62f893-6757df5f6b-vt8nx" podStartSLOduration=1.6329314240000001 podStartE2EDuration="1.632931424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:50:05.631834613 +0000 UTC m=+1445.806436939" watchObservedRunningTime="2026-09-09 10:50:05.632931424 +0000 UTC m=+1445.807533748" Sep 09 10:50:16.622216 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:16.622184 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-62f893-6757df5f6b-vt8nx" Sep 09 10:50:33.902090 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:33.902053 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a03d9e-68676b798c-kbl2x"] Sep 09 10:50:33.905656 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:33.905639 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a03d9e-68676b798c-kbl2x" Sep 09 10:50:33.908457 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:33.908436 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-a03d9e\"" Sep 09 10:50:33.908702 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:33.908687 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-a03d9e-dockercfg-ckk9s\"" Sep 09 10:50:33.917842 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:33.917817 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a03d9e-68676b798c-kbl2x"] Sep 09 10:50:33.961852 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:33.961813 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a03d9e-66f9d5fbb-r2pw8"] Sep 09 10:50:33.965380 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:33.965363 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a03d9e-66f9d5fbb-r2pw8" Sep 09 10:50:33.977613 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:33.977588 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a03d9e-66f9d5fbb-r2pw8"] Sep 09 10:50:34.029554 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:34.029519 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6mpf\" (UniqueName: \"kubernetes.io/projected/adf2f9d1-1c28-4603-84ef-62a908476619-kube-api-access-h6mpf\") pod \"payload-processing-e2e-shared-b-w2-a03d9e-68676b798c-kbl2x\" (UID: \"adf2f9d1-1c28-4603-84ef-62a908476619\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a03d9e-68676b798c-kbl2x" Sep 09 10:50:34.029718 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:34.029565 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbg2l\" (UniqueName: \"kubernetes.io/projected/cafa056b-6f23-4185-b494-cb022cdaa26d-kube-api-access-zbg2l\") pod \"payload-pre-processing-e2e-shared-b-w2-a03d9e-66f9d5fbb-r2pw8\" (UID: \"cafa056b-6f23-4185-b494-cb022cdaa26d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a03d9e-66f9d5fbb-r2pw8" Sep 09 10:50:34.029718 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:34.029661 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf2f9d1-1c28-4603-84ef-62a908476619-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a03d9e-68676b798c-kbl2x\" (UID: \"adf2f9d1-1c28-4603-84ef-62a908476619\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a03d9e-68676b798c-kbl2x" Sep 09 10:50:34.029718 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:34.029695 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cafa056b-6f23-4185-b494-cb022cdaa26d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a03d9e-66f9d5fbb-r2pw8\" (UID: \"cafa056b-6f23-4185-b494-cb022cdaa26d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a03d9e-66f9d5fbb-r2pw8" Sep 09 10:50:34.130930 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:34.130889 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6mpf\" (UniqueName: \"kubernetes.io/projected/adf2f9d1-1c28-4603-84ef-62a908476619-kube-api-access-h6mpf\") pod \"payload-processing-e2e-shared-b-w2-a03d9e-68676b798c-kbl2x\" (UID: \"adf2f9d1-1c28-4603-84ef-62a908476619\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a03d9e-68676b798c-kbl2x" Sep 09 10:50:34.131131 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:34.130940 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbg2l\" (UniqueName: \"kubernetes.io/projected/cafa056b-6f23-4185-b494-cb022cdaa26d-kube-api-access-zbg2l\") pod \"payload-pre-processing-e2e-shared-b-w2-a03d9e-66f9d5fbb-r2pw8\" (UID: \"cafa056b-6f23-4185-b494-cb022cdaa26d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a03d9e-66f9d5fbb-r2pw8" Sep 09 10:50:34.131131 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:34.130978 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf2f9d1-1c28-4603-84ef-62a908476619-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a03d9e-68676b798c-kbl2x\" (UID: \"adf2f9d1-1c28-4603-84ef-62a908476619\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a03d9e-68676b798c-kbl2x" Sep 09 10:50:34.131131 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:34.130997 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cafa056b-6f23-4185-b494-cb022cdaa26d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a03d9e-66f9d5fbb-r2pw8\" (UID: \"cafa056b-6f23-4185-b494-cb022cdaa26d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a03d9e-66f9d5fbb-r2pw8" Sep 09 10:50:34.131715 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:34.131696 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cafa056b-6f23-4185-b494-cb022cdaa26d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a03d9e-66f9d5fbb-r2pw8\" (UID: \"cafa056b-6f23-4185-b494-cb022cdaa26d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a03d9e-66f9d5fbb-r2pw8" Sep 09 10:50:34.131932 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:34.131911 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf2f9d1-1c28-4603-84ef-62a908476619-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a03d9e-68676b798c-kbl2x\" (UID: \"adf2f9d1-1c28-4603-84ef-62a908476619\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a03d9e-68676b798c-kbl2x" Sep 09 10:50:34.143510 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:34.143468 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6mpf\" (UniqueName: \"kubernetes.io/projected/adf2f9d1-1c28-4603-84ef-62a908476619-kube-api-access-h6mpf\") pod \"payload-processing-e2e-shared-b-w2-a03d9e-68676b798c-kbl2x\" (UID: \"adf2f9d1-1c28-4603-84ef-62a908476619\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a03d9e-68676b798c-kbl2x" Sep 09 10:50:34.144091 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:34.144061 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbg2l\" (UniqueName: \"kubernetes.io/projected/cafa056b-6f23-4185-b494-cb022cdaa26d-kube-api-access-zbg2l\") pod \"payload-pre-processing-e2e-shared-b-w2-a03d9e-66f9d5fbb-r2pw8\" (UID: \"cafa056b-6f23-4185-b494-cb022cdaa26d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a03d9e-66f9d5fbb-r2pw8" Sep 09 10:50:34.217290 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:34.217205 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a03d9e-68676b798c-kbl2x" Sep 09 10:50:34.275905 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:34.275651 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a03d9e-66f9d5fbb-r2pw8" Sep 09 10:50:34.497980 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:34.497953 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a03d9e-68676b798c-kbl2x"] Sep 09 10:50:34.499009 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:50:34.498981 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf2f9d1_1c28_4603_84ef_62a908476619.slice/crio-66a73de2068ab1d58651912a66b03b67be0fb9e8ae7852560f47e928c2536e78 WatchSource:0}: Error finding container 66a73de2068ab1d58651912a66b03b67be0fb9e8ae7852560f47e928c2536e78: Status 404 returned error can't find the container with id 66a73de2068ab1d58651912a66b03b67be0fb9e8ae7852560f47e928c2536e78 Sep 09 10:50:34.565244 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:34.565217 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a03d9e-66f9d5fbb-r2pw8"] Sep 09 10:50:34.568081 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:50:34.568050 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcafa056b_6f23_4185_b494_cb022cdaa26d.slice/crio-80a6578459b08af05af3fa53b9b83a5c68a85bd1bbbd9d90d2f20ff95f34a77a WatchSource:0}: Error finding container 80a6578459b08af05af3fa53b9b83a5c68a85bd1bbbd9d90d2f20ff95f34a77a: Status 404 returned error can't find the container with id 80a6578459b08af05af3fa53b9b83a5c68a85bd1bbbd9d90d2f20ff95f34a77a Sep 09 10:50:34.778921 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:34.778823 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a03d9e-68676b798c-kbl2x" event={"ID":"adf2f9d1-1c28-4603-84ef-62a908476619","Type":"ContainerStarted","Data":"598589bea897912cb86f986800c995b4b54d3a4caffa6b252e1b459aff7a7c71"} Sep 09 10:50:34.778921 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:34.778889 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a03d9e-68676b798c-kbl2x" event={"ID":"adf2f9d1-1c28-4603-84ef-62a908476619","Type":"ContainerStarted","Data":"66a73de2068ab1d58651912a66b03b67be0fb9e8ae7852560f47e928c2536e78"} Sep 09 10:50:34.779141 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:34.779018 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a03d9e-68676b798c-kbl2x" Sep 09 10:50:34.780568 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:34.780538 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a03d9e-66f9d5fbb-r2pw8" event={"ID":"cafa056b-6f23-4185-b494-cb022cdaa26d","Type":"ContainerStarted","Data":"bc5d374b55f1e26ec8b292490ec984326aa05f6e5e1388fcd368d74802be14ba"} Sep 09 10:50:34.780722 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:34.780592 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a03d9e-66f9d5fbb-r2pw8" event={"ID":"cafa056b-6f23-4185-b494-cb022cdaa26d","Type":"ContainerStarted","Data":"80a6578459b08af05af3fa53b9b83a5c68a85bd1bbbd9d90d2f20ff95f34a77a"} Sep 09 10:50:34.780722 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:34.780625 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a03d9e-66f9d5fbb-r2pw8" Sep 09 10:50:34.808708 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:34.808662 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a03d9e-68676b798c-kbl2x" podStartSLOduration=1.8086484619999998 podStartE2EDuration="1.808648462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:50:34.806978432 +0000 UTC m=+1474.981580781" watchObservedRunningTime="2026-09-09 10:50:34.808648462 +0000 UTC m=+1474.983250787" Sep 09 10:50:45.787699 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:45.787663 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a03d9e-66f9d5fbb-r2pw8" Sep 09 10:50:45.788096 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:45.787843 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a03d9e-68676b798c-kbl2x" Sep 09 10:50:45.814979 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:50:45.814924 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a03d9e-66f9d5fbb-r2pw8" podStartSLOduration=12.814903054 podStartE2EDuration="12.814903054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:50:34.828793252 +0000 UTC m=+1475.003395576" watchObservedRunningTime="2026-09-09 10:50:45.814903054 +0000 UTC m=+1485.989505379" Sep 09 10:51:00.343137 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:00.343102 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d22ff_0a56b285-3f3c-4f6d-a939-8a95ae00e8da/ovn-acl-logging/0.log" Sep 09 10:51:00.354666 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:00.354630 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d22ff_0a56b285-3f3c-4f6d-a939-8a95ae00e8da/ovn-acl-logging/0.log" Sep 09 10:51:15.195853 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:15.195812 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bbfd5f59-5c557469fc-xmb8v"] Sep 09 10:51:15.199630 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:15.199604 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bbfd5f59-5c557469fc-xmb8v" Sep 09 10:51:15.202438 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:15.202415 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-bbfd5f59\"" Sep 09 10:51:15.202835 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:15.202814 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-bbfd5f59-dockercfg-gbx7h\"" Sep 09 10:51:15.211728 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:15.211699 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bbfd5f59-5c557469fc-xmb8v"] Sep 09 10:51:15.245076 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:15.245038 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bbfd5f59-7c67cb549d-tjkk5"] Sep 09 10:51:15.248843 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:15.248817 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bbfd5f59-7c67cb549d-tjkk5" Sep 09 10:51:15.261964 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:15.261934 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bbfd5f59-7c67cb549d-tjkk5"] Sep 09 10:51:15.307849 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:15.307807 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e455900a-66dc-4716-b940-04bc7d1dab2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bbfd5f59-7c67cb549d-tjkk5\" (UID: \"e455900a-66dc-4716-b940-04bc7d1dab2b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bbfd5f59-7c67cb549d-tjkk5" Sep 09 10:51:15.307849 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:15.307849 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l5ph\" (UniqueName: \"kubernetes.io/projected/e455900a-66dc-4716-b940-04bc7d1dab2b-kube-api-access-2l5ph\") pod \"payload-pre-processing-e2e-mt-bbfd5f59-7c67cb549d-tjkk5\" (UID: \"e455900a-66dc-4716-b940-04bc7d1dab2b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bbfd5f59-7c67cb549d-tjkk5" Sep 09 10:51:15.308083 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:15.307877 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e764eda-3df3-43a0-8a20-febf20979222-plugins-config-volume\") pod \"payload-processing-e2e-mt-bbfd5f59-5c557469fc-xmb8v\" (UID: \"2e764eda-3df3-43a0-8a20-febf20979222\") " pod="openshift-ingress/payload-processing-e2e-mt-bbfd5f59-5c557469fc-xmb8v" Sep 09 10:51:15.308083 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:15.307935 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld9dm\" (UniqueName: \"kubernetes.io/projected/2e764eda-3df3-43a0-8a20-febf20979222-kube-api-access-ld9dm\") pod \"payload-processing-e2e-mt-bbfd5f59-5c557469fc-xmb8v\" (UID: \"2e764eda-3df3-43a0-8a20-febf20979222\") " pod="openshift-ingress/payload-processing-e2e-mt-bbfd5f59-5c557469fc-xmb8v" Sep 09 10:51:15.408679 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:15.408631 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e764eda-3df3-43a0-8a20-febf20979222-plugins-config-volume\") pod \"payload-processing-e2e-mt-bbfd5f59-5c557469fc-xmb8v\" (UID: \"2e764eda-3df3-43a0-8a20-febf20979222\") " pod="openshift-ingress/payload-processing-e2e-mt-bbfd5f59-5c557469fc-xmb8v" Sep 09 10:51:15.408679 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:15.408684 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld9dm\" (UniqueName: \"kubernetes.io/projected/2e764eda-3df3-43a0-8a20-febf20979222-kube-api-access-ld9dm\") pod \"payload-processing-e2e-mt-bbfd5f59-5c557469fc-xmb8v\" (UID: \"2e764eda-3df3-43a0-8a20-febf20979222\") " pod="openshift-ingress/payload-processing-e2e-mt-bbfd5f59-5c557469fc-xmb8v" Sep 09 10:51:15.408955 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:15.408764 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e455900a-66dc-4716-b940-04bc7d1dab2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bbfd5f59-7c67cb549d-tjkk5\" (UID: \"e455900a-66dc-4716-b940-04bc7d1dab2b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bbfd5f59-7c67cb549d-tjkk5" Sep 09 10:51:15.408955 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:15.408785 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l5ph\" (UniqueName: \"kubernetes.io/projected/e455900a-66dc-4716-b940-04bc7d1dab2b-kube-api-access-2l5ph\") pod \"payload-pre-processing-e2e-mt-bbfd5f59-7c67cb549d-tjkk5\" (UID: \"e455900a-66dc-4716-b940-04bc7d1dab2b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bbfd5f59-7c67cb549d-tjkk5" Sep 09 10:51:15.409389 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:15.409361 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e764eda-3df3-43a0-8a20-febf20979222-plugins-config-volume\") pod \"payload-processing-e2e-mt-bbfd5f59-5c557469fc-xmb8v\" (UID: \"2e764eda-3df3-43a0-8a20-febf20979222\") " pod="openshift-ingress/payload-processing-e2e-mt-bbfd5f59-5c557469fc-xmb8v" Sep 09 10:51:15.409589 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:15.409541 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e455900a-66dc-4716-b940-04bc7d1dab2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bbfd5f59-7c67cb549d-tjkk5\" (UID: \"e455900a-66dc-4716-b940-04bc7d1dab2b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bbfd5f59-7c67cb549d-tjkk5" Sep 09 10:51:15.418279 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:15.418244 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l5ph\" (UniqueName: \"kubernetes.io/projected/e455900a-66dc-4716-b940-04bc7d1dab2b-kube-api-access-2l5ph\") pod \"payload-pre-processing-e2e-mt-bbfd5f59-7c67cb549d-tjkk5\" (UID: \"e455900a-66dc-4716-b940-04bc7d1dab2b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bbfd5f59-7c67cb549d-tjkk5" Sep 09 10:51:15.418279 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:15.418274 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld9dm\" (UniqueName: \"kubernetes.io/projected/2e764eda-3df3-43a0-8a20-febf20979222-kube-api-access-ld9dm\") pod \"payload-processing-e2e-mt-bbfd5f59-5c557469fc-xmb8v\" (UID: \"2e764eda-3df3-43a0-8a20-febf20979222\") " pod="openshift-ingress/payload-processing-e2e-mt-bbfd5f59-5c557469fc-xmb8v" Sep 09 10:51:15.511213 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:15.511117 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bbfd5f59-5c557469fc-xmb8v" Sep 09 10:51:15.560494 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:15.560106 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bbfd5f59-7c67cb549d-tjkk5" Sep 09 10:51:15.786276 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:15.786068 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bbfd5f59-5c557469fc-xmb8v"] Sep 09 10:51:15.788620 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:51:15.788596 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e764eda_3df3_43a0_8a20_febf20979222.slice/crio-657505eac9e5893dc270738397518ed352cd1058d7c3e51f44d6ad37dc30b2f4 WatchSource:0}: Error finding container 657505eac9e5893dc270738397518ed352cd1058d7c3e51f44d6ad37dc30b2f4: Status 404 returned error can't find the container with id 657505eac9e5893dc270738397518ed352cd1058d7c3e51f44d6ad37dc30b2f4 Sep 09 10:51:15.856281 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:15.856255 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bbfd5f59-7c67cb549d-tjkk5"] Sep 09 10:51:15.859044 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:51:15.859019 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode455900a_66dc_4716_b940_04bc7d1dab2b.slice/crio-8d4843a18b939fc6215e1204852dd0746447c5e14af4cfccfe6d4fe55080a7a1 WatchSource:0}: Error finding container 8d4843a18b939fc6215e1204852dd0746447c5e14af4cfccfe6d4fe55080a7a1: Status 404 returned error can't find the container with id 8d4843a18b939fc6215e1204852dd0746447c5e14af4cfccfe6d4fe55080a7a1 Sep 09 10:51:16.016124 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:16.016085 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bbfd5f59-7c67cb549d-tjkk5" event={"ID":"e455900a-66dc-4716-b940-04bc7d1dab2b","Type":"ContainerStarted","Data":"3b59d386620b59e1e61044f9476330fc91375dbdd6d6a3be268529a629982d5a"} Sep 09 10:51:16.016315 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:16.016154 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bbfd5f59-7c67cb549d-tjkk5" event={"ID":"e455900a-66dc-4716-b940-04bc7d1dab2b","Type":"ContainerStarted","Data":"8d4843a18b939fc6215e1204852dd0746447c5e14af4cfccfe6d4fe55080a7a1"} Sep 09 10:51:16.016315 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:16.016181 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-bbfd5f59-7c67cb549d-tjkk5" Sep 09 10:51:16.017777 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:16.017751 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bbfd5f59-5c557469fc-xmb8v" event={"ID":"2e764eda-3df3-43a0-8a20-febf20979222","Type":"ContainerStarted","Data":"f3fe83c283b2165571662dceb604ed9aade076f7ebc59178a611a34743fd91ac"} Sep 09 10:51:16.017921 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:16.017789 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bbfd5f59-5c557469fc-xmb8v" event={"ID":"2e764eda-3df3-43a0-8a20-febf20979222","Type":"ContainerStarted","Data":"657505eac9e5893dc270738397518ed352cd1058d7c3e51f44d6ad37dc30b2f4"} Sep 09 10:51:16.017921 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:16.017871 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-bbfd5f59-5c557469fc-xmb8v" Sep 09 10:51:16.033316 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:16.033262 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-bbfd5f59-7c67cb549d-tjkk5" podStartSLOduration=1.033245496 podStartE2EDuration="1.033245496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:51:16.032790264 +0000 UTC m=+1516.207392591" watchObservedRunningTime="2026-09-09 10:51:16.033245496 +0000 UTC m=+1516.207847821" Sep 09 10:51:16.050014 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:16.049902 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-bbfd5f59-5c557469fc-xmb8v" podStartSLOduration=1.049879218 podStartE2EDuration="1.049879218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:51:16.048641274 +0000 UTC m=+1516.223243601" watchObservedRunningTime="2026-09-09 10:51:16.049879218 +0000 UTC m=+1516.224481544" Sep 09 10:51:27.026085 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:27.026049 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-bbfd5f59-5c557469fc-xmb8v" Sep 09 10:51:27.028387 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:51:27.026119 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-bbfd5f59-7c67cb549d-tjkk5" Sep 09 10:52:00.439099 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:00.439066 2559 scope.go:117] "RemoveContainer" containerID="7598fc17d5d476ea2585ab4f6f923adbba2a9393a124964a300b79c19d9df6a6" Sep 09 10:52:00.449274 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:00.449255 2559 scope.go:117] "RemoveContainer" containerID="24e7abcb3783c4d877f8e627b9898ee6bd6fbbb0dc351f58bf3f7f2ed24dd0d8" Sep 09 10:52:00.457355 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:00.457334 2559 scope.go:117] "RemoveContainer" containerID="166af4109bd0d3e0fcc05ff2520448b1a710ed93f3473e288591c4e32ffcc6ee" Sep 09 10:52:38.335439 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:38.335408 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv"] Sep 09 10:52:38.435203 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:38.435165 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bbfd5f59-vwvmw"] Sep 09 10:52:38.438897 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:38.438870 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bbfd5f59-vwvmw" Sep 09 10:52:38.441718 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:38.441694 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 10:52:38.441945 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:38.441726 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kq7qv\"" Sep 09 10:52:38.450529 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:38.450479 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bbfd5f59-vwvmw"] Sep 09 10:52:38.629991 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:38.629898 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0334512e-93d6-45e7-b63e-b5b47cdc2933-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bbfd5f59-vwvmw\" (UID: \"0334512e-93d6-45e7-b63e-b5b47cdc2933\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bbfd5f59-vwvmw" Sep 09 10:52:38.730808 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:38.730774 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0334512e-93d6-45e7-b63e-b5b47cdc2933-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bbfd5f59-vwvmw\" (UID: \"0334512e-93d6-45e7-b63e-b5b47cdc2933\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bbfd5f59-vwvmw" Sep 09 10:52:38.731389 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:38.731367 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0334512e-93d6-45e7-b63e-b5b47cdc2933-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bbfd5f59-vwvmw\" (UID: \"0334512e-93d6-45e7-b63e-b5b47cdc2933\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bbfd5f59-vwvmw" Sep 09 10:52:38.752776 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:38.752742 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bbfd5f59-vwvmw" Sep 09 10:52:39.026166 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:39.026141 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bbfd5f59-vwvmw"] Sep 09 10:52:39.028693 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:52:39.028657 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0334512e_93d6_45e7_b63e_b5b47cdc2933.slice/crio-caefc0dd848dfe09a3eed0f176ddc062fc5e4027b5486dc762f291ac640d6170 WatchSource:0}: Error finding container caefc0dd848dfe09a3eed0f176ddc062fc5e4027b5486dc762f291ac640d6170: Status 404 returned error can't find the container with id caefc0dd848dfe09a3eed0f176ddc062fc5e4027b5486dc762f291ac640d6170 Sep 09 10:52:39.471886 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:39.471855 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bbfd5f59-vwvmw" event={"ID":"0334512e-93d6-45e7-b63e-b5b47cdc2933","Type":"ContainerStarted","Data":"55f5cc71d3d4fa782b9a19c09c39f21242c9a4ffb9b0f76ba95ab1d9b19adcc1"} Sep 09 10:52:39.472283 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:39.471904 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bbfd5f59-vwvmw" event={"ID":"0334512e-93d6-45e7-b63e-b5b47cdc2933","Type":"ContainerStarted","Data":"caefc0dd848dfe09a3eed0f176ddc062fc5e4027b5486dc762f291ac640d6170"} Sep 09 10:52:39.488667 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:39.488615 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bbfd5f59-vwvmw" podStartSLOduration=1.48859804 podStartE2EDuration="1.48859804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:52:39.488209428 +0000 UTC m=+1599.662811754" watchObservedRunningTime="2026-09-09 10:52:39.48859804 +0000 UTC m=+1599.663200414" Sep 09 10:52:40.478855 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:40.478823 2559 generic.go:356] "Generic (PLEG): container finished" podID="0334512e-93d6-45e7-b63e-b5b47cdc2933" containerID="55f5cc71d3d4fa782b9a19c09c39f21242c9a4ffb9b0f76ba95ab1d9b19adcc1" exitCode=0 Sep 09 10:52:40.479199 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:40.478933 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bbfd5f59-vwvmw" event={"ID":"0334512e-93d6-45e7-b63e-b5b47cdc2933","Type":"ContainerDied","Data":"55f5cc71d3d4fa782b9a19c09c39f21242c9a4ffb9b0f76ba95ab1d9b19adcc1"} Sep 09 10:52:41.616367 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:41.616342 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bbfd5f59-vwvmw" Sep 09 10:52:41.654457 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:41.654405 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0334512e-93d6-45e7-b63e-b5b47cdc2933-maas-api-service-ca\") pod \"0334512e-93d6-45e7-b63e-b5b47cdc2933\" (UID: \"0334512e-93d6-45e7-b63e-b5b47cdc2933\") " Sep 09 10:52:41.654831 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:41.654803 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0334512e-93d6-45e7-b63e-b5b47cdc2933-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0334512e-93d6-45e7-b63e-b5b47cdc2933" (UID: "0334512e-93d6-45e7-b63e-b5b47cdc2933"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:52:41.755338 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:41.755233 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0334512e-93d6-45e7-b63e-b5b47cdc2933-maas-api-service-ca\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:52:42.491646 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:42.491613 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bbfd5f59-vwvmw" Sep 09 10:52:42.491827 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:42.491613 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bbfd5f59-vwvmw" event={"ID":"0334512e-93d6-45e7-b63e-b5b47cdc2933","Type":"ContainerDied","Data":"caefc0dd848dfe09a3eed0f176ddc062fc5e4027b5486dc762f291ac640d6170"} Sep 09 10:52:42.491827 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:42.491747 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="caefc0dd848dfe09a3eed0f176ddc062fc5e4027b5486dc762f291ac640d6170" Sep 09 10:52:48.908490 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:48.908456 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv"] Sep 09 10:52:48.908924 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:48.908898 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0334512e-93d6-45e7-b63e-b5b47cdc2933" containerName="revoke-keys" Sep 09 10:52:48.908924 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:48.908917 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0334512e-93d6-45e7-b63e-b5b47cdc2933" containerName="revoke-keys" Sep 09 10:52:48.909110 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:48.909026 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0334512e-93d6-45e7-b63e-b5b47cdc2933" containerName="revoke-keys" Sep 09 10:52:48.912303 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:48.912284 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" Sep 09 10:52:48.914906 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:48.914883 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1eb21d\"/\"sub-test-model-1eb21d-kserve-self-signed-certs\"" Sep 09 10:52:48.921134 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:48.921111 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv"] Sep 09 10:52:49.006731 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:49.006691 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cde86a70-a35e-44d1-8b51-22d03a3d2750-tmp-dir\") pod \"sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv\" (UID: \"cde86a70-a35e-44d1-8b51-22d03a3d2750\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" Sep 09 10:52:49.006943 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:49.006743 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cde86a70-a35e-44d1-8b51-22d03a3d2750-home\") pod \"sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv\" (UID: \"cde86a70-a35e-44d1-8b51-22d03a3d2750\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" Sep 09 10:52:49.006943 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:49.006760 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde86a70-a35e-44d1-8b51-22d03a3d2750-tls-certs\") pod \"sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv\" (UID: \"cde86a70-a35e-44d1-8b51-22d03a3d2750\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" Sep 09 10:52:49.006943 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:49.006778 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cde86a70-a35e-44d1-8b51-22d03a3d2750-model-cache\") pod \"sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv\" (UID: \"cde86a70-a35e-44d1-8b51-22d03a3d2750\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" Sep 09 10:52:49.006943 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:49.006897 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vptl7\" (UniqueName: \"kubernetes.io/projected/cde86a70-a35e-44d1-8b51-22d03a3d2750-kube-api-access-vptl7\") pod \"sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv\" (UID: \"cde86a70-a35e-44d1-8b51-22d03a3d2750\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" Sep 09 10:52:49.006943 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:49.006941 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cde86a70-a35e-44d1-8b51-22d03a3d2750-dshm\") pod \"sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv\" (UID: \"cde86a70-a35e-44d1-8b51-22d03a3d2750\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" Sep 09 10:52:49.107905 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:49.107549 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vptl7\" (UniqueName: \"kubernetes.io/projected/cde86a70-a35e-44d1-8b51-22d03a3d2750-kube-api-access-vptl7\") pod \"sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv\" (UID: \"cde86a70-a35e-44d1-8b51-22d03a3d2750\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" Sep 09 10:52:49.108224 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:49.108199 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cde86a70-a35e-44d1-8b51-22d03a3d2750-dshm\") pod \"sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv\" (UID: \"cde86a70-a35e-44d1-8b51-22d03a3d2750\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" Sep 09 10:52:49.108814 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:49.108793 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cde86a70-a35e-44d1-8b51-22d03a3d2750-tmp-dir\") pod \"sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv\" (UID: \"cde86a70-a35e-44d1-8b51-22d03a3d2750\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" Sep 09 10:52:49.109403 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:49.109282 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cde86a70-a35e-44d1-8b51-22d03a3d2750-tmp-dir\") pod \"sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv\" (UID: \"cde86a70-a35e-44d1-8b51-22d03a3d2750\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" Sep 09 10:52:49.110385 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:49.109608 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cde86a70-a35e-44d1-8b51-22d03a3d2750-home\") pod \"sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv\" (UID: \"cde86a70-a35e-44d1-8b51-22d03a3d2750\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" Sep 09 10:52:49.110385 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:49.109648 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde86a70-a35e-44d1-8b51-22d03a3d2750-tls-certs\") pod \"sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv\" (UID: \"cde86a70-a35e-44d1-8b51-22d03a3d2750\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" Sep 09 10:52:49.110385 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:49.109679 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cde86a70-a35e-44d1-8b51-22d03a3d2750-model-cache\") pod \"sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv\" (UID: \"cde86a70-a35e-44d1-8b51-22d03a3d2750\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" Sep 09 10:52:49.110385 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:49.110019 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cde86a70-a35e-44d1-8b51-22d03a3d2750-model-cache\") pod \"sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv\" (UID: \"cde86a70-a35e-44d1-8b51-22d03a3d2750\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" Sep 09 10:52:49.110385 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:49.110123 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cde86a70-a35e-44d1-8b51-22d03a3d2750-home\") pod \"sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv\" (UID: \"cde86a70-a35e-44d1-8b51-22d03a3d2750\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" Sep 09 10:52:49.111863 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:49.111837 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cde86a70-a35e-44d1-8b51-22d03a3d2750-dshm\") pod \"sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv\" (UID: \"cde86a70-a35e-44d1-8b51-22d03a3d2750\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" Sep 09 10:52:49.113522 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:49.113464 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde86a70-a35e-44d1-8b51-22d03a3d2750-tls-certs\") pod \"sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv\" (UID: \"cde86a70-a35e-44d1-8b51-22d03a3d2750\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" Sep 09 10:52:49.119766 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:49.119729 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vptl7\" (UniqueName: \"kubernetes.io/projected/cde86a70-a35e-44d1-8b51-22d03a3d2750-kube-api-access-vptl7\") pod \"sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv\" (UID: \"cde86a70-a35e-44d1-8b51-22d03a3d2750\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" Sep 09 10:52:49.223285 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:49.223187 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" Sep 09 10:52:49.503247 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:49.503218 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv"] Sep 09 10:52:49.507311 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:52:49.506274 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcde86a70_a35e_44d1_8b51_22d03a3d2750.slice/crio-a41a7905a19c9572daa29a17e7afc30f23cd0215d9aa360da81710f7aa5e8a1b WatchSource:0}: Error finding container a41a7905a19c9572daa29a17e7afc30f23cd0215d9aa360da81710f7aa5e8a1b: Status 404 returned error can't find the container with id a41a7905a19c9572daa29a17e7afc30f23cd0215d9aa360da81710f7aa5e8a1b Sep 09 10:52:49.530544 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:49.530515 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" event={"ID":"cde86a70-a35e-44d1-8b51-22d03a3d2750","Type":"ContainerStarted","Data":"a41a7905a19c9572daa29a17e7afc30f23cd0215d9aa360da81710f7aa5e8a1b"} Sep 09 10:52:50.537806 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:50.537766 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" event={"ID":"cde86a70-a35e-44d1-8b51-22d03a3d2750","Type":"ContainerStarted","Data":"7f9decb4816612a489d0eb95b8730663e7ea14b10400ee2e2b987756cca70117"} Sep 09 10:52:50.555602 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:50.555540 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" podStartSLOduration=2.555521626 podStartE2EDuration="2.555521626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:52:50.555278116 +0000 UTC m=+1610.729880444" watchObservedRunningTime="2026-09-09 10:52:50.555521626 +0000 UTC m=+1610.730123945" Sep 09 10:52:53.355771 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:53.355725 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" podUID="9f0820db-9a67-4291-ac82-d4040621ed03" containerName="main" containerID="cri-o://0f33bc0327720d72e80c7e2d3bbb0e1b27239fdc7df77c7e0bf877038385e7a2" gracePeriod=45 Sep 09 10:52:53.558695 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:53.558644 2559 generic.go:356] "Generic (PLEG): container finished" podID="9f0820db-9a67-4291-ac82-d4040621ed03" containerID="0f33bc0327720d72e80c7e2d3bbb0e1b27239fdc7df77c7e0bf877038385e7a2" exitCode=0 Sep 09 10:52:53.558860 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:53.558715 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" event={"ID":"9f0820db-9a67-4291-ac82-d4040621ed03","Type":"ContainerDied","Data":"0f33bc0327720d72e80c7e2d3bbb0e1b27239fdc7df77c7e0bf877038385e7a2"} Sep 09 10:52:53.619651 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:53.619587 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" Sep 09 10:52:53.646758 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:53.646719 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f0820db-9a67-4291-ac82-d4040621ed03-tls-certs\") pod \"9f0820db-9a67-4291-ac82-d4040621ed03\" (UID: \"9f0820db-9a67-4291-ac82-d4040621ed03\") " Sep 09 10:52:53.646940 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:53.646801 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f0820db-9a67-4291-ac82-d4040621ed03-model-cache\") pod \"9f0820db-9a67-4291-ac82-d4040621ed03\" (UID: \"9f0820db-9a67-4291-ac82-d4040621ed03\") " Sep 09 10:52:53.646940 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:53.646858 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ms7lp\" (UniqueName: \"kubernetes.io/projected/9f0820db-9a67-4291-ac82-d4040621ed03-kube-api-access-ms7lp\") pod \"9f0820db-9a67-4291-ac82-d4040621ed03\" (UID: \"9f0820db-9a67-4291-ac82-d4040621ed03\") " Sep 09 10:52:53.646940 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:53.646885 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f0820db-9a67-4291-ac82-d4040621ed03-tmp-dir\") pod \"9f0820db-9a67-4291-ac82-d4040621ed03\" (UID: \"9f0820db-9a67-4291-ac82-d4040621ed03\") " Sep 09 10:52:53.646940 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:53.646941 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f0820db-9a67-4291-ac82-d4040621ed03-home\") pod \"9f0820db-9a67-4291-ac82-d4040621ed03\" (UID: \"9f0820db-9a67-4291-ac82-d4040621ed03\") " Sep 09 10:52:53.647168 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:53.646971 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f0820db-9a67-4291-ac82-d4040621ed03-dshm\") pod \"9f0820db-9a67-4291-ac82-d4040621ed03\" (UID: \"9f0820db-9a67-4291-ac82-d4040621ed03\") " Sep 09 10:52:53.647949 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:53.647882 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0820db-9a67-4291-ac82-d4040621ed03-home" (OuterVolumeSpecName: "home") pod "9f0820db-9a67-4291-ac82-d4040621ed03" (UID: "9f0820db-9a67-4291-ac82-d4040621ed03"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 10:52:53.648383 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:53.648181 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0820db-9a67-4291-ac82-d4040621ed03-model-cache" (OuterVolumeSpecName: "model-cache") pod "9f0820db-9a67-4291-ac82-d4040621ed03" (UID: "9f0820db-9a67-4291-ac82-d4040621ed03"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 10:52:53.648626 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:53.648582 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0820db-9a67-4291-ac82-d4040621ed03-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9f0820db-9a67-4291-ac82-d4040621ed03" (UID: "9f0820db-9a67-4291-ac82-d4040621ed03"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 10:52:53.649726 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:53.649703 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0820db-9a67-4291-ac82-d4040621ed03-dshm" (OuterVolumeSpecName: "dshm") pod "9f0820db-9a67-4291-ac82-d4040621ed03" (UID: "9f0820db-9a67-4291-ac82-d4040621ed03"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 10:52:53.650146 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:53.650127 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f0820db-9a67-4291-ac82-d4040621ed03-kube-api-access-ms7lp" (OuterVolumeSpecName: "kube-api-access-ms7lp") pod "9f0820db-9a67-4291-ac82-d4040621ed03" (UID: "9f0820db-9a67-4291-ac82-d4040621ed03"). InnerVolumeSpecName "kube-api-access-ms7lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:52:53.650395 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:53.650368 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f0820db-9a67-4291-ac82-d4040621ed03-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9f0820db-9a67-4291-ac82-d4040621ed03" (UID: "9f0820db-9a67-4291-ac82-d4040621ed03"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:52:53.747951 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:53.747875 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f0820db-9a67-4291-ac82-d4040621ed03-model-cache\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:52:53.747951 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:53.747905 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms7lp\" (UniqueName: \"kubernetes.io/projected/9f0820db-9a67-4291-ac82-d4040621ed03-kube-api-access-ms7lp\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:52:53.747951 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:53.747915 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f0820db-9a67-4291-ac82-d4040621ed03-tmp-dir\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:52:53.747951 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:53.747925 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f0820db-9a67-4291-ac82-d4040621ed03-home\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:52:53.747951 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:53.747932 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f0820db-9a67-4291-ac82-d4040621ed03-dshm\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:52:53.747951 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:53.747941 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f0820db-9a67-4291-ac82-d4040621ed03-tls-certs\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:52:54.236662 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.236619 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bbfd5f59-5c557469fc-xmb8v"] Sep 09 10:52:54.236988 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.236949 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-bbfd5f59-5c557469fc-xmb8v" podUID="2e764eda-3df3-43a0-8a20-febf20979222" containerName="payload-processing" containerID="cri-o://f3fe83c283b2165571662dceb604ed9aade076f7ebc59178a611a34743fd91ac" gracePeriod=30 Sep 09 10:52:54.242007 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.241975 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bbfd5f59-7c67cb549d-tjkk5"] Sep 09 10:52:54.242269 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.242241 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-bbfd5f59-7c67cb549d-tjkk5" podUID="e455900a-66dc-4716-b940-04bc7d1dab2b" containerName="payload-pre-processing" containerID="cri-o://3b59d386620b59e1e61044f9476330fc91375dbdd6d6a3be268529a629982d5a" gracePeriod=30 Sep 09 10:52:54.473687 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.473651 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bbfd5f59-vwvmw"] Sep 09 10:52:54.480059 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.480026 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bbfd5f59-vwvmw"] Sep 09 10:52:54.539632 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.539606 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bbfd5f59-5c557469fc-xmb8v" Sep 09 10:52:54.542384 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.542359 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bbfd5f59-7c67cb549d-tjkk5" Sep 09 10:52:54.555766 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.555732 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e455900a-66dc-4716-b940-04bc7d1dab2b-plugins-config-volume\") pod \"e455900a-66dc-4716-b940-04bc7d1dab2b\" (UID: \"e455900a-66dc-4716-b940-04bc7d1dab2b\") " Sep 09 10:52:54.555938 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.555803 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2l5ph\" (UniqueName: \"kubernetes.io/projected/e455900a-66dc-4716-b940-04bc7d1dab2b-kube-api-access-2l5ph\") pod \"e455900a-66dc-4716-b940-04bc7d1dab2b\" (UID: \"e455900a-66dc-4716-b940-04bc7d1dab2b\") " Sep 09 10:52:54.555938 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.555858 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ld9dm\" (UniqueName: \"kubernetes.io/projected/2e764eda-3df3-43a0-8a20-febf20979222-kube-api-access-ld9dm\") pod \"2e764eda-3df3-43a0-8a20-febf20979222\" (UID: \"2e764eda-3df3-43a0-8a20-febf20979222\") " Sep 09 10:52:54.555938 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.555878 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e764eda-3df3-43a0-8a20-febf20979222-plugins-config-volume\") pod \"2e764eda-3df3-43a0-8a20-febf20979222\" (UID: \"2e764eda-3df3-43a0-8a20-febf20979222\") " Sep 09 10:52:54.556193 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.556165 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e455900a-66dc-4716-b940-04bc7d1dab2b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e455900a-66dc-4716-b940-04bc7d1dab2b" (UID: "e455900a-66dc-4716-b940-04bc7d1dab2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:52:54.556338 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.556316 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e764eda-3df3-43a0-8a20-febf20979222-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2e764eda-3df3-43a0-8a20-febf20979222" (UID: "2e764eda-3df3-43a0-8a20-febf20979222"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:52:54.558731 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.558703 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e455900a-66dc-4716-b940-04bc7d1dab2b-kube-api-access-2l5ph" (OuterVolumeSpecName: "kube-api-access-2l5ph") pod "e455900a-66dc-4716-b940-04bc7d1dab2b" (UID: "e455900a-66dc-4716-b940-04bc7d1dab2b"). InnerVolumeSpecName "kube-api-access-2l5ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:52:54.558888 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.558867 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e764eda-3df3-43a0-8a20-febf20979222-kube-api-access-ld9dm" (OuterVolumeSpecName: "kube-api-access-ld9dm") pod "2e764eda-3df3-43a0-8a20-febf20979222" (UID: "2e764eda-3df3-43a0-8a20-febf20979222"). InnerVolumeSpecName "kube-api-access-ld9dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:52:54.567812 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.567778 2559 generic.go:356] "Generic (PLEG): container finished" podID="2e764eda-3df3-43a0-8a20-febf20979222" containerID="f3fe83c283b2165571662dceb604ed9aade076f7ebc59178a611a34743fd91ac" exitCode=0 Sep 09 10:52:54.567934 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.567842 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bbfd5f59-5c557469fc-xmb8v" event={"ID":"2e764eda-3df3-43a0-8a20-febf20979222","Type":"ContainerDied","Data":"f3fe83c283b2165571662dceb604ed9aade076f7ebc59178a611a34743fd91ac"} Sep 09 10:52:54.567934 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.567906 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bbfd5f59-5c557469fc-xmb8v" event={"ID":"2e764eda-3df3-43a0-8a20-febf20979222","Type":"ContainerDied","Data":"657505eac9e5893dc270738397518ed352cd1058d7c3e51f44d6ad37dc30b2f4"} Sep 09 10:52:54.568050 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.567936 2559 scope.go:117] "RemoveContainer" containerID="f3fe83c283b2165571662dceb604ed9aade076f7ebc59178a611a34743fd91ac" Sep 09 10:52:54.568153 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.568135 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bbfd5f59-5c557469fc-xmb8v" Sep 09 10:52:54.570794 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.570727 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" event={"ID":"9f0820db-9a67-4291-ac82-d4040621ed03","Type":"ContainerDied","Data":"387237739d5cf34152f11f53ea8b342ad7c812f92e455bce4851cd9385590e20"} Sep 09 10:52:54.570911 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.570874 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv" Sep 09 10:52:54.572908 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.572880 2559 generic.go:356] "Generic (PLEG): container finished" podID="e455900a-66dc-4716-b940-04bc7d1dab2b" containerID="3b59d386620b59e1e61044f9476330fc91375dbdd6d6a3be268529a629982d5a" exitCode=0 Sep 09 10:52:54.573038 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.572924 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bbfd5f59-7c67cb549d-tjkk5" Sep 09 10:52:54.573038 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.572988 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bbfd5f59-7c67cb549d-tjkk5" event={"ID":"e455900a-66dc-4716-b940-04bc7d1dab2b","Type":"ContainerDied","Data":"3b59d386620b59e1e61044f9476330fc91375dbdd6d6a3be268529a629982d5a"} Sep 09 10:52:54.573468 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.573037 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bbfd5f59-7c67cb549d-tjkk5" event={"ID":"e455900a-66dc-4716-b940-04bc7d1dab2b","Type":"ContainerDied","Data":"8d4843a18b939fc6215e1204852dd0746447c5e14af4cfccfe6d4fe55080a7a1"} Sep 09 10:52:54.586197 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.586169 2559 scope.go:117] "RemoveContainer" containerID="f3fe83c283b2165571662dceb604ed9aade076f7ebc59178a611a34743fd91ac" Sep 09 10:52:54.586567 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:52:54.586543 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3fe83c283b2165571662dceb604ed9aade076f7ebc59178a611a34743fd91ac\": container with ID starting with f3fe83c283b2165571662dceb604ed9aade076f7ebc59178a611a34743fd91ac not found: ID does not exist" containerID="f3fe83c283b2165571662dceb604ed9aade076f7ebc59178a611a34743fd91ac" Sep 09 10:52:54.586677 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.586574 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3fe83c283b2165571662dceb604ed9aade076f7ebc59178a611a34743fd91ac"} err="failed to get container status \"f3fe83c283b2165571662dceb604ed9aade076f7ebc59178a611a34743fd91ac\": rpc error: code = NotFound desc = could not find container \"f3fe83c283b2165571662dceb604ed9aade076f7ebc59178a611a34743fd91ac\": container with ID starting with f3fe83c283b2165571662dceb604ed9aade076f7ebc59178a611a34743fd91ac not found: ID does not exist" Sep 09 10:52:54.586677 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.586593 2559 scope.go:117] "RemoveContainer" containerID="0f33bc0327720d72e80c7e2d3bbb0e1b27239fdc7df77c7e0bf877038385e7a2" Sep 09 10:52:54.596451 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.596430 2559 scope.go:117] "RemoveContainer" containerID="3b59d386620b59e1e61044f9476330fc91375dbdd6d6a3be268529a629982d5a" Sep 09 10:52:54.604134 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.604106 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv"] Sep 09 10:52:54.605796 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.605774 2559 scope.go:117] "RemoveContainer" containerID="3b59d386620b59e1e61044f9476330fc91375dbdd6d6a3be268529a629982d5a" Sep 09 10:52:54.606140 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:52:54.606117 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b59d386620b59e1e61044f9476330fc91375dbdd6d6a3be268529a629982d5a\": container with ID starting with 3b59d386620b59e1e61044f9476330fc91375dbdd6d6a3be268529a629982d5a not found: ID does not exist" containerID="3b59d386620b59e1e61044f9476330fc91375dbdd6d6a3be268529a629982d5a" Sep 09 10:52:54.606219 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.606144 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b59d386620b59e1e61044f9476330fc91375dbdd6d6a3be268529a629982d5a"} err="failed to get container status \"3b59d386620b59e1e61044f9476330fc91375dbdd6d6a3be268529a629982d5a\": rpc error: code = NotFound desc = could not find container \"3b59d386620b59e1e61044f9476330fc91375dbdd6d6a3be268529a629982d5a\": container with ID starting with 3b59d386620b59e1e61044f9476330fc91375dbdd6d6a3be268529a629982d5a not found: ID does not exist" Sep 09 10:52:54.610753 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.610731 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1eb21d/auth-test-model-1eb21d-kserve-6d5c4cdc4c-b2vhv"] Sep 09 10:52:54.620293 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.620268 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bbfd5f59-7c67cb549d-tjkk5"] Sep 09 10:52:54.626693 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.626667 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bbfd5f59-7c67cb549d-tjkk5"] Sep 09 10:52:54.638776 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.638750 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bbfd5f59-5c557469fc-xmb8v"] Sep 09 10:52:54.644670 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.644633 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bbfd5f59-5c557469fc-xmb8v"] Sep 09 10:52:54.657418 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.657395 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e764eda-3df3-43a0-8a20-febf20979222-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:52:54.657418 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.657419 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e455900a-66dc-4716-b940-04bc7d1dab2b-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:52:54.657598 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.657431 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l5ph\" (UniqueName: \"kubernetes.io/projected/e455900a-66dc-4716-b940-04bc7d1dab2b-kube-api-access-2l5ph\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:52:54.657598 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:54.657440 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld9dm\" (UniqueName: \"kubernetes.io/projected/2e764eda-3df3-43a0-8a20-febf20979222-kube-api-access-ld9dm\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:52:56.334026 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:56.333992 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0334512e-93d6-45e7-b63e-b5b47cdc2933" path="/var/lib/kubelet/pods/0334512e-93d6-45e7-b63e-b5b47cdc2933/volumes" Sep 09 10:52:56.334391 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:56.334348 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e764eda-3df3-43a0-8a20-febf20979222" path="/var/lib/kubelet/pods/2e764eda-3df3-43a0-8a20-febf20979222/volumes" Sep 09 10:52:56.334737 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:56.334723 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f0820db-9a67-4291-ac82-d4040621ed03" path="/var/lib/kubelet/pods/9f0820db-9a67-4291-ac82-d4040621ed03/volumes" Sep 09 10:52:56.335035 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:56.335024 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e455900a-66dc-4716-b940-04bc7d1dab2b" path="/var/lib/kubelet/pods/e455900a-66dc-4716-b940-04bc7d1dab2b/volumes" Sep 09 10:52:59.223388 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:59.223344 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" Sep 09 10:52:59.223812 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:59.223643 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" Sep 09 10:52:59.236618 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:59.236589 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" Sep 09 10:52:59.613414 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:52:59.613386 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" Sep 09 10:53:00.485430 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:53:00.485391 2559 scope.go:117] "RemoveContainer" containerID="d6290ded53fcddc43798d310e3fd72a0198df7dc6a8c3da44d766f25cba86174" Sep 09 10:54:00.518852 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:00.518808 2559 scope.go:117] "RemoveContainer" containerID="83936076cd166efe6fbc77f4c5b4147a8655f5078a04470eb3842723a3f98300" Sep 09 10:54:12.598380 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:12.598345 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv"] Sep 09 10:54:27.618351 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:27.618244 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" podUID="cde86a70-a35e-44d1-8b51-22d03a3d2750" containerName="main" containerID="cri-o://7f9decb4816612a489d0eb95b8730663e7ea14b10400ee2e2b987756cca70117" gracePeriod=45 Sep 09 10:54:27.874055 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:27.873992 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" Sep 09 10:54:27.892310 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:27.892274 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vptl7\" (UniqueName: \"kubernetes.io/projected/cde86a70-a35e-44d1-8b51-22d03a3d2750-kube-api-access-vptl7\") pod \"cde86a70-a35e-44d1-8b51-22d03a3d2750\" (UID: \"cde86a70-a35e-44d1-8b51-22d03a3d2750\") " Sep 09 10:54:27.892492 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:27.892321 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cde86a70-a35e-44d1-8b51-22d03a3d2750-model-cache\") pod \"cde86a70-a35e-44d1-8b51-22d03a3d2750\" (UID: \"cde86a70-a35e-44d1-8b51-22d03a3d2750\") " Sep 09 10:54:27.892492 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:27.892348 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cde86a70-a35e-44d1-8b51-22d03a3d2750-dshm\") pod \"cde86a70-a35e-44d1-8b51-22d03a3d2750\" (UID: \"cde86a70-a35e-44d1-8b51-22d03a3d2750\") " Sep 09 10:54:27.892492 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:27.892398 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cde86a70-a35e-44d1-8b51-22d03a3d2750-tmp-dir\") pod \"cde86a70-a35e-44d1-8b51-22d03a3d2750\" (UID: \"cde86a70-a35e-44d1-8b51-22d03a3d2750\") " Sep 09 10:54:27.892492 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:27.892437 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde86a70-a35e-44d1-8b51-22d03a3d2750-tls-certs\") pod \"cde86a70-a35e-44d1-8b51-22d03a3d2750\" (UID: \"cde86a70-a35e-44d1-8b51-22d03a3d2750\") " Sep 09 10:54:27.892492 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:27.892470 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cde86a70-a35e-44d1-8b51-22d03a3d2750-home\") pod \"cde86a70-a35e-44d1-8b51-22d03a3d2750\" (UID: \"cde86a70-a35e-44d1-8b51-22d03a3d2750\") " Sep 09 10:54:27.892780 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:27.892614 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde86a70-a35e-44d1-8b51-22d03a3d2750-model-cache" (OuterVolumeSpecName: "model-cache") pod "cde86a70-a35e-44d1-8b51-22d03a3d2750" (UID: "cde86a70-a35e-44d1-8b51-22d03a3d2750"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 10:54:27.892780 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:27.892633 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde86a70-a35e-44d1-8b51-22d03a3d2750-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cde86a70-a35e-44d1-8b51-22d03a3d2750" (UID: "cde86a70-a35e-44d1-8b51-22d03a3d2750"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 10:54:27.892890 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:27.892800 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde86a70-a35e-44d1-8b51-22d03a3d2750-home" (OuterVolumeSpecName: "home") pod "cde86a70-a35e-44d1-8b51-22d03a3d2750" (UID: "cde86a70-a35e-44d1-8b51-22d03a3d2750"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 10:54:27.892890 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:27.892854 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cde86a70-a35e-44d1-8b51-22d03a3d2750-model-cache\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:54:27.892890 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:27.892872 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cde86a70-a35e-44d1-8b51-22d03a3d2750-tmp-dir\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:54:27.894680 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:27.894652 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cde86a70-a35e-44d1-8b51-22d03a3d2750-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cde86a70-a35e-44d1-8b51-22d03a3d2750" (UID: "cde86a70-a35e-44d1-8b51-22d03a3d2750"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:54:27.895335 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:27.894942 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde86a70-a35e-44d1-8b51-22d03a3d2750-dshm" (OuterVolumeSpecName: "dshm") pod "cde86a70-a35e-44d1-8b51-22d03a3d2750" (UID: "cde86a70-a35e-44d1-8b51-22d03a3d2750"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 10:54:27.895335 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:27.895187 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde86a70-a35e-44d1-8b51-22d03a3d2750-kube-api-access-vptl7" (OuterVolumeSpecName: "kube-api-access-vptl7") pod "cde86a70-a35e-44d1-8b51-22d03a3d2750" (UID: "cde86a70-a35e-44d1-8b51-22d03a3d2750"). InnerVolumeSpecName "kube-api-access-vptl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:54:27.994165 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:27.994123 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde86a70-a35e-44d1-8b51-22d03a3d2750-tls-certs\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:54:27.994165 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:27.994159 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cde86a70-a35e-44d1-8b51-22d03a3d2750-home\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:54:27.994165 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:27.994169 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vptl7\" (UniqueName: \"kubernetes.io/projected/cde86a70-a35e-44d1-8b51-22d03a3d2750-kube-api-access-vptl7\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:54:27.994396 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:27.994179 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cde86a70-a35e-44d1-8b51-22d03a3d2750-dshm\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:54:28.074272 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:28.074231 2559 generic.go:356] "Generic (PLEG): container finished" podID="cde86a70-a35e-44d1-8b51-22d03a3d2750" containerID="7f9decb4816612a489d0eb95b8730663e7ea14b10400ee2e2b987756cca70117" exitCode=0 Sep 09 10:54:28.074272 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:28.074276 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" Sep 09 10:54:28.074470 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:28.074316 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" event={"ID":"cde86a70-a35e-44d1-8b51-22d03a3d2750","Type":"ContainerDied","Data":"7f9decb4816612a489d0eb95b8730663e7ea14b10400ee2e2b987756cca70117"} Sep 09 10:54:28.074470 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:28.074362 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv" event={"ID":"cde86a70-a35e-44d1-8b51-22d03a3d2750","Type":"ContainerDied","Data":"a41a7905a19c9572daa29a17e7afc30f23cd0215d9aa360da81710f7aa5e8a1b"} Sep 09 10:54:28.074470 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:28.074380 2559 scope.go:117] "RemoveContainer" containerID="7f9decb4816612a489d0eb95b8730663e7ea14b10400ee2e2b987756cca70117" Sep 09 10:54:28.083312 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:28.083293 2559 scope.go:117] "RemoveContainer" containerID="7f9decb4816612a489d0eb95b8730663e7ea14b10400ee2e2b987756cca70117" Sep 09 10:54:28.083596 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:54:28.083575 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f9decb4816612a489d0eb95b8730663e7ea14b10400ee2e2b987756cca70117\": container with ID starting with 7f9decb4816612a489d0eb95b8730663e7ea14b10400ee2e2b987756cca70117 not found: ID does not exist" containerID="7f9decb4816612a489d0eb95b8730663e7ea14b10400ee2e2b987756cca70117" Sep 09 10:54:28.083659 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:28.083605 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f9decb4816612a489d0eb95b8730663e7ea14b10400ee2e2b987756cca70117"} err="failed to get container status \"7f9decb4816612a489d0eb95b8730663e7ea14b10400ee2e2b987756cca70117\": rpc error: code = NotFound desc = could not find container \"7f9decb4816612a489d0eb95b8730663e7ea14b10400ee2e2b987756cca70117\": container with ID starting with 7f9decb4816612a489d0eb95b8730663e7ea14b10400ee2e2b987756cca70117 not found: ID does not exist" Sep 09 10:54:28.095698 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:28.095674 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv"] Sep 09 10:54:28.102232 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:28.102211 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1eb21d/sub-test-model-1eb21d-kserve-f47bbb5b9-86rrv"] Sep 09 10:54:28.334926 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:28.334895 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cde86a70-a35e-44d1-8b51-22d03a3d2750" path="/var/lib/kubelet/pods/cde86a70-a35e-44d1-8b51-22d03a3d2750/volumes" Sep 09 10:54:29.504464 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.504433 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2"] Sep 09 10:54:29.504867 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.504843 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f0820db-9a67-4291-ac82-d4040621ed03" containerName="main" Sep 09 10:54:29.504867 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.504856 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f0820db-9a67-4291-ac82-d4040621ed03" containerName="main" Sep 09 10:54:29.504867 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.504863 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde86a70-a35e-44d1-8b51-22d03a3d2750" containerName="main" Sep 09 10:54:29.504867 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.504869 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde86a70-a35e-44d1-8b51-22d03a3d2750" containerName="main" Sep 09 10:54:29.505006 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.504876 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e764eda-3df3-43a0-8a20-febf20979222" containerName="payload-processing" Sep 09 10:54:29.505006 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.504882 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e764eda-3df3-43a0-8a20-febf20979222" containerName="payload-processing" Sep 09 10:54:29.505006 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.504897 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e455900a-66dc-4716-b940-04bc7d1dab2b" containerName="payload-pre-processing" Sep 09 10:54:29.505006 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.504902 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e455900a-66dc-4716-b940-04bc7d1dab2b" containerName="payload-pre-processing" Sep 09 10:54:29.505006 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.504966 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cde86a70-a35e-44d1-8b51-22d03a3d2750" containerName="main" Sep 09 10:54:29.505006 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.504977 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e764eda-3df3-43a0-8a20-febf20979222" containerName="payload-processing" Sep 09 10:54:29.505006 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.504982 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e455900a-66dc-4716-b940-04bc7d1dab2b" containerName="payload-pre-processing" Sep 09 10:54:29.505006 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.504990 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f0820db-9a67-4291-ac82-d4040621ed03" containerName="main" Sep 09 10:54:29.509276 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.509254 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" Sep 09 10:54:29.512064 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.512046 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1eb21d\"/\"openshift-service-ca.crt\"" Sep 09 10:54:29.513175 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.513153 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1eb21d\"/\"default-dockercfg-kmqqb\"" Sep 09 10:54:29.513316 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.513191 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1eb21d\"/\"kube-root-ca.crt\"" Sep 09 10:54:29.513584 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.513204 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1eb21d\"/\"rate-test-model-1eb21d-kserve-self-signed-certs\"" Sep 09 10:54:29.519869 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.519843 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2"] Sep 09 10:54:29.607280 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.607243 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02cd273e-83d2-4c79-a636-ab2b33e46117-home\") pod \"rate-test-model-1eb21d-kserve-f768b65bc-wgdv2\" (UID: \"02cd273e-83d2-4c79-a636-ab2b33e46117\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" Sep 09 10:54:29.607459 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.607301 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg8j8\" (UniqueName: \"kubernetes.io/projected/02cd273e-83d2-4c79-a636-ab2b33e46117-kube-api-access-dg8j8\") pod \"rate-test-model-1eb21d-kserve-f768b65bc-wgdv2\" (UID: \"02cd273e-83d2-4c79-a636-ab2b33e46117\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" Sep 09 10:54:29.607459 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.607320 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02cd273e-83d2-4c79-a636-ab2b33e46117-tmp-dir\") pod \"rate-test-model-1eb21d-kserve-f768b65bc-wgdv2\" (UID: \"02cd273e-83d2-4c79-a636-ab2b33e46117\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" Sep 09 10:54:29.607459 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.607353 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02cd273e-83d2-4c79-a636-ab2b33e46117-dshm\") pod \"rate-test-model-1eb21d-kserve-f768b65bc-wgdv2\" (UID: \"02cd273e-83d2-4c79-a636-ab2b33e46117\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" Sep 09 10:54:29.607459 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.607369 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02cd273e-83d2-4c79-a636-ab2b33e46117-tls-certs\") pod \"rate-test-model-1eb21d-kserve-f768b65bc-wgdv2\" (UID: \"02cd273e-83d2-4c79-a636-ab2b33e46117\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" Sep 09 10:54:29.607459 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.607387 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02cd273e-83d2-4c79-a636-ab2b33e46117-model-cache\") pod \"rate-test-model-1eb21d-kserve-f768b65bc-wgdv2\" (UID: \"02cd273e-83d2-4c79-a636-ab2b33e46117\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" Sep 09 10:54:29.708252 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.708208 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg8j8\" (UniqueName: \"kubernetes.io/projected/02cd273e-83d2-4c79-a636-ab2b33e46117-kube-api-access-dg8j8\") pod \"rate-test-model-1eb21d-kserve-f768b65bc-wgdv2\" (UID: \"02cd273e-83d2-4c79-a636-ab2b33e46117\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" Sep 09 10:54:29.708252 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.708252 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02cd273e-83d2-4c79-a636-ab2b33e46117-tmp-dir\") pod \"rate-test-model-1eb21d-kserve-f768b65bc-wgdv2\" (UID: \"02cd273e-83d2-4c79-a636-ab2b33e46117\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" Sep 09 10:54:29.708488 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.708295 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02cd273e-83d2-4c79-a636-ab2b33e46117-dshm\") pod \"rate-test-model-1eb21d-kserve-f768b65bc-wgdv2\" (UID: \"02cd273e-83d2-4c79-a636-ab2b33e46117\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" Sep 09 10:54:29.708488 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.708312 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02cd273e-83d2-4c79-a636-ab2b33e46117-tls-certs\") pod \"rate-test-model-1eb21d-kserve-f768b65bc-wgdv2\" (UID: \"02cd273e-83d2-4c79-a636-ab2b33e46117\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" Sep 09 10:54:29.708488 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.708334 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02cd273e-83d2-4c79-a636-ab2b33e46117-model-cache\") pod \"rate-test-model-1eb21d-kserve-f768b65bc-wgdv2\" (UID: \"02cd273e-83d2-4c79-a636-ab2b33e46117\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" Sep 09 10:54:29.708488 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.708374 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02cd273e-83d2-4c79-a636-ab2b33e46117-home\") pod \"rate-test-model-1eb21d-kserve-f768b65bc-wgdv2\" (UID: \"02cd273e-83d2-4c79-a636-ab2b33e46117\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" Sep 09 10:54:29.708870 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.708842 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02cd273e-83d2-4c79-a636-ab2b33e46117-tmp-dir\") pod \"rate-test-model-1eb21d-kserve-f768b65bc-wgdv2\" (UID: \"02cd273e-83d2-4c79-a636-ab2b33e46117\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" Sep 09 10:54:29.708947 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.708857 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02cd273e-83d2-4c79-a636-ab2b33e46117-model-cache\") pod \"rate-test-model-1eb21d-kserve-f768b65bc-wgdv2\" (UID: \"02cd273e-83d2-4c79-a636-ab2b33e46117\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" Sep 09 10:54:29.708947 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.708882 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02cd273e-83d2-4c79-a636-ab2b33e46117-home\") pod \"rate-test-model-1eb21d-kserve-f768b65bc-wgdv2\" (UID: \"02cd273e-83d2-4c79-a636-ab2b33e46117\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" Sep 09 10:54:29.710804 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.710768 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02cd273e-83d2-4c79-a636-ab2b33e46117-dshm\") pod \"rate-test-model-1eb21d-kserve-f768b65bc-wgdv2\" (UID: \"02cd273e-83d2-4c79-a636-ab2b33e46117\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" Sep 09 10:54:29.710927 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.710903 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02cd273e-83d2-4c79-a636-ab2b33e46117-tls-certs\") pod \"rate-test-model-1eb21d-kserve-f768b65bc-wgdv2\" (UID: \"02cd273e-83d2-4c79-a636-ab2b33e46117\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" Sep 09 10:54:29.717032 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.717009 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg8j8\" (UniqueName: \"kubernetes.io/projected/02cd273e-83d2-4c79-a636-ab2b33e46117-kube-api-access-dg8j8\") pod \"rate-test-model-1eb21d-kserve-f768b65bc-wgdv2\" (UID: \"02cd273e-83d2-4c79-a636-ab2b33e46117\") " pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" Sep 09 10:54:29.821609 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:29.821486 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" Sep 09 10:54:30.099674 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:30.099650 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2"] Sep 09 10:54:30.101721 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:54:30.101695 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02cd273e_83d2_4c79_a636_ab2b33e46117.slice/crio-4299ca973daf0eeb1cbe17c58b791cdec6922b58e741c18def0d195acfaef2fc WatchSource:0}: Error finding container 4299ca973daf0eeb1cbe17c58b791cdec6922b58e741c18def0d195acfaef2fc: Status 404 returned error can't find the container with id 4299ca973daf0eeb1cbe17c58b791cdec6922b58e741c18def0d195acfaef2fc Sep 09 10:54:31.092957 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:31.092919 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" event={"ID":"02cd273e-83d2-4c79-a636-ab2b33e46117","Type":"ContainerStarted","Data":"e3cf662d74c75475c27046f2785b8a6d00c8089c2ccf54ad9c87bc921b4bdd1d"} Sep 09 10:54:31.093362 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:31.092967 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" event={"ID":"02cd273e-83d2-4c79-a636-ab2b33e46117","Type":"ContainerStarted","Data":"4299ca973daf0eeb1cbe17c58b791cdec6922b58e741c18def0d195acfaef2fc"} Sep 09 10:54:31.114161 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:31.114107 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" podStartSLOduration=2.114090472 podStartE2EDuration="2.114090472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:54:31.111859213 +0000 UTC m=+1711.286461564" watchObservedRunningTime="2026-09-09 10:54:31.114090472 +0000 UTC m=+1711.288692798" Sep 09 10:54:39.821831 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:39.821796 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" Sep 09 10:54:39.822243 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:39.821865 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" Sep 09 10:54:39.834374 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:39.834344 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" Sep 09 10:54:40.149034 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:40.149001 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" Sep 09 10:54:47.659961 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:47.659925 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e5f9c5-8985b6d54-5wgqc"] Sep 09 10:54:47.663441 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:47.663422 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e5f9c5-8985b6d54-5wgqc" Sep 09 10:54:47.666441 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:47.666418 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e5f9c5\"" Sep 09 10:54:47.666618 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:47.666600 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e5f9c5-dockercfg-9lqs6\"" Sep 09 10:54:47.677324 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:47.677295 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e5f9c5-8985b6d54-5wgqc"] Sep 09 10:54:47.754148 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:47.754111 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e5f9c5-64b65fcc45-jg2l7"] Sep 09 10:54:47.758277 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:47.758249 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e5f9c5-64b65fcc45-jg2l7" Sep 09 10:54:47.773072 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:47.773040 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e5f9c5-64b65fcc45-jg2l7"] Sep 09 10:54:47.775794 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:47.775764 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0947dc1-afc5-4358-8f25-622c94493529-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e5f9c5-8985b6d54-5wgqc\" (UID: \"b0947dc1-afc5-4358-8f25-622c94493529\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e5f9c5-8985b6d54-5wgqc" Sep 09 10:54:47.775939 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:47.775849 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ttlt\" (UniqueName: \"kubernetes.io/projected/b0947dc1-afc5-4358-8f25-622c94493529-kube-api-access-9ttlt\") pod \"payload-pre-processing-e2e-api-a-e5f9c5-8985b6d54-5wgqc\" (UID: \"b0947dc1-afc5-4358-8f25-622c94493529\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e5f9c5-8985b6d54-5wgqc" Sep 09 10:54:47.876483 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:47.876444 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a7a18d-8aab-4b7f-8bdb-68bec973a43a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e5f9c5-64b65fcc45-jg2l7\" (UID: \"b9a7a18d-8aab-4b7f-8bdb-68bec973a43a\") " pod="openshift-ingress/payload-processing-e2e-api-a-e5f9c5-64b65fcc45-jg2l7" Sep 09 10:54:47.876709 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:47.876492 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0947dc1-afc5-4358-8f25-622c94493529-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e5f9c5-8985b6d54-5wgqc\" (UID: \"b0947dc1-afc5-4358-8f25-622c94493529\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e5f9c5-8985b6d54-5wgqc" Sep 09 10:54:47.876709 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:47.876635 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ttlt\" (UniqueName: \"kubernetes.io/projected/b0947dc1-afc5-4358-8f25-622c94493529-kube-api-access-9ttlt\") pod \"payload-pre-processing-e2e-api-a-e5f9c5-8985b6d54-5wgqc\" (UID: \"b0947dc1-afc5-4358-8f25-622c94493529\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e5f9c5-8985b6d54-5wgqc" Sep 09 10:54:47.876709 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:47.876676 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj9ms\" (UniqueName: \"kubernetes.io/projected/b9a7a18d-8aab-4b7f-8bdb-68bec973a43a-kube-api-access-nj9ms\") pod \"payload-processing-e2e-api-a-e5f9c5-64b65fcc45-jg2l7\" (UID: \"b9a7a18d-8aab-4b7f-8bdb-68bec973a43a\") " pod="openshift-ingress/payload-processing-e2e-api-a-e5f9c5-64b65fcc45-jg2l7" Sep 09 10:54:47.877295 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:47.877277 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0947dc1-afc5-4358-8f25-622c94493529-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e5f9c5-8985b6d54-5wgqc\" (UID: \"b0947dc1-afc5-4358-8f25-622c94493529\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e5f9c5-8985b6d54-5wgqc" Sep 09 10:54:47.886407 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:47.886377 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ttlt\" (UniqueName: \"kubernetes.io/projected/b0947dc1-afc5-4358-8f25-622c94493529-kube-api-access-9ttlt\") pod \"payload-pre-processing-e2e-api-a-e5f9c5-8985b6d54-5wgqc\" (UID: \"b0947dc1-afc5-4358-8f25-622c94493529\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e5f9c5-8985b6d54-5wgqc" Sep 09 10:54:47.974270 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:47.974175 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e5f9c5-8985b6d54-5wgqc" Sep 09 10:54:47.977666 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:47.977163 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a7a18d-8aab-4b7f-8bdb-68bec973a43a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e5f9c5-64b65fcc45-jg2l7\" (UID: \"b9a7a18d-8aab-4b7f-8bdb-68bec973a43a\") " pod="openshift-ingress/payload-processing-e2e-api-a-e5f9c5-64b65fcc45-jg2l7" Sep 09 10:54:47.977666 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:47.977211 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj9ms\" (UniqueName: \"kubernetes.io/projected/b9a7a18d-8aab-4b7f-8bdb-68bec973a43a-kube-api-access-nj9ms\") pod \"payload-processing-e2e-api-a-e5f9c5-64b65fcc45-jg2l7\" (UID: \"b9a7a18d-8aab-4b7f-8bdb-68bec973a43a\") " pod="openshift-ingress/payload-processing-e2e-api-a-e5f9c5-64b65fcc45-jg2l7" Sep 09 10:54:47.977845 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:47.977823 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a7a18d-8aab-4b7f-8bdb-68bec973a43a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e5f9c5-64b65fcc45-jg2l7\" (UID: \"b9a7a18d-8aab-4b7f-8bdb-68bec973a43a\") " pod="openshift-ingress/payload-processing-e2e-api-a-e5f9c5-64b65fcc45-jg2l7" Sep 09 10:54:47.986005 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:47.985978 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj9ms\" (UniqueName: \"kubernetes.io/projected/b9a7a18d-8aab-4b7f-8bdb-68bec973a43a-kube-api-access-nj9ms\") pod \"payload-processing-e2e-api-a-e5f9c5-64b65fcc45-jg2l7\" (UID: \"b9a7a18d-8aab-4b7f-8bdb-68bec973a43a\") " pod="openshift-ingress/payload-processing-e2e-api-a-e5f9c5-64b65fcc45-jg2l7" Sep 09 10:54:48.071848 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:48.071786 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e5f9c5-64b65fcc45-jg2l7" Sep 09 10:54:48.251134 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:48.251104 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e5f9c5-8985b6d54-5wgqc"] Sep 09 10:54:48.252926 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:54:48.252903 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0947dc1_afc5_4358_8f25_622c94493529.slice/crio-f96255630480102a480580e64f87830107febfec06bb56ec2628b104b90998df WatchSource:0}: Error finding container f96255630480102a480580e64f87830107febfec06bb56ec2628b104b90998df: Status 404 returned error can't find the container with id f96255630480102a480580e64f87830107febfec06bb56ec2628b104b90998df Sep 09 10:54:48.349303 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:48.349268 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e5f9c5-64b65fcc45-jg2l7"] Sep 09 10:54:48.351584 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:54:48.351553 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a7a18d_8aab_4b7f_8bdb_68bec973a43a.slice/crio-553a478e906a4112adc6c4a63e165df2ad20eeb9db3ee0c3eb9c94d5ce364bab WatchSource:0}: Error finding container 553a478e906a4112adc6c4a63e165df2ad20eeb9db3ee0c3eb9c94d5ce364bab: Status 404 returned error can't find the container with id 553a478e906a4112adc6c4a63e165df2ad20eeb9db3ee0c3eb9c94d5ce364bab Sep 09 10:54:49.190173 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:49.190121 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e5f9c5-64b65fcc45-jg2l7" event={"ID":"b9a7a18d-8aab-4b7f-8bdb-68bec973a43a","Type":"ContainerStarted","Data":"2d64e59b2dcd504504e93bb2c7c590e4a5be3c3c1af5157e468316b809aede43"} Sep 09 10:54:49.190657 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:49.190187 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e5f9c5-64b65fcc45-jg2l7" event={"ID":"b9a7a18d-8aab-4b7f-8bdb-68bec973a43a","Type":"ContainerStarted","Data":"553a478e906a4112adc6c4a63e165df2ad20eeb9db3ee0c3eb9c94d5ce364bab"} Sep 09 10:54:49.190657 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:49.190232 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-e5f9c5-64b65fcc45-jg2l7" Sep 09 10:54:49.191964 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:49.191937 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e5f9c5-8985b6d54-5wgqc" event={"ID":"b0947dc1-afc5-4358-8f25-622c94493529","Type":"ContainerStarted","Data":"8845f22801240dff0dc65955641dbd9a535976689586a65b33173bc035682d0c"} Sep 09 10:54:49.192090 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:49.191982 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e5f9c5-8985b6d54-5wgqc" event={"ID":"b0947dc1-afc5-4358-8f25-622c94493529","Type":"ContainerStarted","Data":"f96255630480102a480580e64f87830107febfec06bb56ec2628b104b90998df"} Sep 09 10:54:49.192090 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:49.192004 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e5f9c5-8985b6d54-5wgqc" Sep 09 10:54:49.211374 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:49.211313 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-e5f9c5-64b65fcc45-jg2l7" podStartSLOduration=2.211291366 podStartE2EDuration="2.211291366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:54:49.208752152 +0000 UTC m=+1729.383354477" watchObservedRunningTime="2026-09-09 10:54:49.211291366 +0000 UTC m=+1729.385893691" Sep 09 10:54:49.228381 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:54:49.228324 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e5f9c5-8985b6d54-5wgqc" podStartSLOduration=2.22830574 podStartE2EDuration="2.22830574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:54:49.225760481 +0000 UTC m=+1729.400362805" watchObservedRunningTime="2026-09-09 10:54:49.22830574 +0000 UTC m=+1729.402908067" Sep 09 10:55:00.200044 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:00.200008 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e5f9c5-8985b6d54-5wgqc" Sep 09 10:55:00.200526 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:00.200228 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-e5f9c5-64b65fcc45-jg2l7" Sep 09 10:55:00.539141 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:00.539060 2559 scope.go:117] "RemoveContainer" containerID="d55c998bcde81ff56f841f3e715f799414be739ac20812f7f9660ec3291488c0" Sep 09 10:55:00.548054 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:00.548030 2559 scope.go:117] "RemoveContainer" containerID="7d4cab799ba5f7a431b2c38796ff7c09063a6775b931cd7ebd2ff49339d9f160" Sep 09 10:55:00.555881 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:00.555854 2559 scope.go:117] "RemoveContainer" containerID="1959afb334def63288a376ff4d0a78fd56334c36e73d3d14c135dd08f1799cbd" Sep 09 10:55:16.590569 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:16.590521 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-798524-5bcf6b74c-k7frc"] Sep 09 10:55:16.594052 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:16.594028 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-798524-5bcf6b74c-k7frc" Sep 09 10:55:16.596759 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:16.596736 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-798524\"" Sep 09 10:55:16.596879 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:16.596737 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-798524-dockercfg-bp4f8\"" Sep 09 10:55:16.605111 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:16.605086 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-798524-5bcf6b74c-k7frc"] Sep 09 10:55:16.633784 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:16.633738 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5afa4a78-e5b7-4108-8dba-b1a36fa133d6-plugins-config-volume\") pod \"payload-processing-e2e-api-b-798524-5bcf6b74c-k7frc\" (UID: \"5afa4a78-e5b7-4108-8dba-b1a36fa133d6\") " pod="openshift-ingress/payload-processing-e2e-api-b-798524-5bcf6b74c-k7frc" Sep 09 10:55:16.634001 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:16.633873 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df6mb\" (UniqueName: \"kubernetes.io/projected/5afa4a78-e5b7-4108-8dba-b1a36fa133d6-kube-api-access-df6mb\") pod \"payload-processing-e2e-api-b-798524-5bcf6b74c-k7frc\" (UID: \"5afa4a78-e5b7-4108-8dba-b1a36fa133d6\") " pod="openshift-ingress/payload-processing-e2e-api-b-798524-5bcf6b74c-k7frc" Sep 09 10:55:16.735398 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:16.735350 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5afa4a78-e5b7-4108-8dba-b1a36fa133d6-plugins-config-volume\") pod \"payload-processing-e2e-api-b-798524-5bcf6b74c-k7frc\" (UID: \"5afa4a78-e5b7-4108-8dba-b1a36fa133d6\") " pod="openshift-ingress/payload-processing-e2e-api-b-798524-5bcf6b74c-k7frc" Sep 09 10:55:16.735644 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:16.735545 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df6mb\" (UniqueName: \"kubernetes.io/projected/5afa4a78-e5b7-4108-8dba-b1a36fa133d6-kube-api-access-df6mb\") pod \"payload-processing-e2e-api-b-798524-5bcf6b74c-k7frc\" (UID: \"5afa4a78-e5b7-4108-8dba-b1a36fa133d6\") " pod="openshift-ingress/payload-processing-e2e-api-b-798524-5bcf6b74c-k7frc" Sep 09 10:55:16.736108 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:16.736088 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5afa4a78-e5b7-4108-8dba-b1a36fa133d6-plugins-config-volume\") pod \"payload-processing-e2e-api-b-798524-5bcf6b74c-k7frc\" (UID: \"5afa4a78-e5b7-4108-8dba-b1a36fa133d6\") " pod="openshift-ingress/payload-processing-e2e-api-b-798524-5bcf6b74c-k7frc" Sep 09 10:55:16.744846 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:16.744818 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df6mb\" (UniqueName: \"kubernetes.io/projected/5afa4a78-e5b7-4108-8dba-b1a36fa133d6-kube-api-access-df6mb\") pod \"payload-processing-e2e-api-b-798524-5bcf6b74c-k7frc\" (UID: \"5afa4a78-e5b7-4108-8dba-b1a36fa133d6\") " pod="openshift-ingress/payload-processing-e2e-api-b-798524-5bcf6b74c-k7frc" Sep 09 10:55:16.904947 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:16.904853 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-798524-5bcf6b74c-k7frc" Sep 09 10:55:17.184751 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:17.184658 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-798524-5bcf6b74c-k7frc"] Sep 09 10:55:17.188017 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:55:17.187986 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5afa4a78_e5b7_4108_8dba_b1a36fa133d6.slice/crio-bda5690702a062de19e5c1beeb87d8d7fe8f7bff01c5b94fa74cbbb65fd67aaa WatchSource:0}: Error finding container bda5690702a062de19e5c1beeb87d8d7fe8f7bff01c5b94fa74cbbb65fd67aaa: Status 404 returned error can't find the container with id bda5690702a062de19e5c1beeb87d8d7fe8f7bff01c5b94fa74cbbb65fd67aaa Sep 09 10:55:17.336136 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:17.336095 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-798524-5bcf6b74c-k7frc" event={"ID":"5afa4a78-e5b7-4108-8dba-b1a36fa133d6","Type":"ContainerStarted","Data":"bb49fe3bd29b634aae76bfc23fa6a3b78d0ad7e2c53c33b2751491534f43dc0d"} Sep 09 10:55:17.336325 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:17.336154 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-798524-5bcf6b74c-k7frc" event={"ID":"5afa4a78-e5b7-4108-8dba-b1a36fa133d6","Type":"ContainerStarted","Data":"bda5690702a062de19e5c1beeb87d8d7fe8f7bff01c5b94fa74cbbb65fd67aaa"} Sep 09 10:55:17.336325 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:17.336256 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-798524-5bcf6b74c-k7frc" Sep 09 10:55:17.362346 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:17.362285 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-798524-5bcf6b74c-k7frc" podStartSLOduration=1.362266089 podStartE2EDuration="1.362266089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:55:17.360196462 +0000 UTC m=+1757.534798786" watchObservedRunningTime="2026-09-09 10:55:17.362266089 +0000 UTC m=+1757.536868442" Sep 09 10:55:28.342970 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:28.342927 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-798524-5bcf6b74c-k7frc" Sep 09 10:55:42.836802 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:42.836766 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e5f9c5-rbmcj"] Sep 09 10:55:42.840209 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:42.840192 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e5f9c5-rbmcj" Sep 09 10:55:42.851883 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:42.851857 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 10:55:42.852155 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:42.852139 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kq7qv\"" Sep 09 10:55:42.869487 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:42.869450 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e5f9c5-rbmcj"] Sep 09 10:55:42.989309 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:42.989265 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dab98061-a35b-4a6e-816d-3e0262d66767-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e5f9c5-rbmcj\" (UID: \"dab98061-a35b-4a6e-816d-3e0262d66767\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e5f9c5-rbmcj" Sep 09 10:55:43.090953 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:43.090851 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dab98061-a35b-4a6e-816d-3e0262d66767-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e5f9c5-rbmcj\" (UID: \"dab98061-a35b-4a6e-816d-3e0262d66767\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e5f9c5-rbmcj" Sep 09 10:55:43.097020 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:43.092049 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dab98061-a35b-4a6e-816d-3e0262d66767-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e5f9c5-rbmcj\" (UID: \"dab98061-a35b-4a6e-816d-3e0262d66767\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e5f9c5-rbmcj" Sep 09 10:55:43.150598 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:43.150547 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e5f9c5-rbmcj" Sep 09 10:55:43.424162 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:43.424064 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e5f9c5-rbmcj"] Sep 09 10:55:43.427297 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:55:43.427268 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab98061_a35b_4a6e_816d_3e0262d66767.slice/crio-b6272aeaceb9168f162cfb3f180afde2f9799f28a9ce13a2091e3dff95355b31 WatchSource:0}: Error finding container b6272aeaceb9168f162cfb3f180afde2f9799f28a9ce13a2091e3dff95355b31: Status 404 returned error can't find the container with id b6272aeaceb9168f162cfb3f180afde2f9799f28a9ce13a2091e3dff95355b31 Sep 09 10:55:43.471572 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:43.471532 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e5f9c5-rbmcj" event={"ID":"dab98061-a35b-4a6e-816d-3e0262d66767","Type":"ContainerStarted","Data":"b6272aeaceb9168f162cfb3f180afde2f9799f28a9ce13a2091e3dff95355b31"} Sep 09 10:55:44.478658 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:44.478620 2559 generic.go:356] "Generic (PLEG): container finished" podID="dab98061-a35b-4a6e-816d-3e0262d66767" containerID="5a06433ea37fbe297f37b9978c1fc1ac514e95cf57cf41562cd21753b5dc5329" exitCode=0 Sep 09 10:55:44.479033 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:44.478705 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e5f9c5-rbmcj" event={"ID":"dab98061-a35b-4a6e-816d-3e0262d66767","Type":"ContainerDied","Data":"5a06433ea37fbe297f37b9978c1fc1ac514e95cf57cf41562cd21753b5dc5329"} Sep 09 10:55:45.626954 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:45.626920 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e5f9c5-rbmcj" Sep 09 10:55:45.816315 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:45.816221 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dab98061-a35b-4a6e-816d-3e0262d66767-maas-api-service-ca\") pod \"dab98061-a35b-4a6e-816d-3e0262d66767\" (UID: \"dab98061-a35b-4a6e-816d-3e0262d66767\") " Sep 09 10:55:45.816603 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:45.816579 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dab98061-a35b-4a6e-816d-3e0262d66767-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dab98061-a35b-4a6e-816d-3e0262d66767" (UID: "dab98061-a35b-4a6e-816d-3e0262d66767"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:55:45.917109 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:45.917066 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dab98061-a35b-4a6e-816d-3e0262d66767-maas-api-service-ca\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:55:46.490971 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:46.490946 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e5f9c5-rbmcj" Sep 09 10:55:46.491124 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:46.490979 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e5f9c5-rbmcj" event={"ID":"dab98061-a35b-4a6e-816d-3e0262d66767","Type":"ContainerDied","Data":"b6272aeaceb9168f162cfb3f180afde2f9799f28a9ce13a2091e3dff95355b31"} Sep 09 10:55:46.491124 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:46.491023 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6272aeaceb9168f162cfb3f180afde2f9799f28a9ce13a2091e3dff95355b31" Sep 09 10:55:58.431394 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:58.431350 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e5f9c5-64b65fcc45-jg2l7"] Sep 09 10:55:58.431837 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:58.431635 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-e5f9c5-64b65fcc45-jg2l7" podUID="b9a7a18d-8aab-4b7f-8bdb-68bec973a43a" containerName="payload-processing" containerID="cri-o://2d64e59b2dcd504504e93bb2c7c590e4a5be3c3c1af5157e468316b809aede43" gracePeriod=30 Sep 09 10:55:58.438307 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:58.438038 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e5f9c5-8985b6d54-5wgqc"] Sep 09 10:55:58.438307 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:58.438272 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e5f9c5-8985b6d54-5wgqc" podUID="b0947dc1-afc5-4358-8f25-622c94493529" containerName="payload-pre-processing" containerID="cri-o://8845f22801240dff0dc65955641dbd9a535976689586a65b33173bc035682d0c" gracePeriod=30 Sep 09 10:55:58.559636 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:58.559606 2559 generic.go:356] "Generic (PLEG): container finished" podID="b9a7a18d-8aab-4b7f-8bdb-68bec973a43a" containerID="2d64e59b2dcd504504e93bb2c7c590e4a5be3c3c1af5157e468316b809aede43" exitCode=0 Sep 09 10:55:58.559819 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:58.559674 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e5f9c5-64b65fcc45-jg2l7" event={"ID":"b9a7a18d-8aab-4b7f-8bdb-68bec973a43a","Type":"ContainerDied","Data":"2d64e59b2dcd504504e93bb2c7c590e4a5be3c3c1af5157e468316b809aede43"} Sep 09 10:55:58.561418 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:58.561392 2559 generic.go:356] "Generic (PLEG): container finished" podID="b0947dc1-afc5-4358-8f25-622c94493529" containerID="8845f22801240dff0dc65955641dbd9a535976689586a65b33173bc035682d0c" exitCode=0 Sep 09 10:55:58.561552 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:58.561464 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e5f9c5-8985b6d54-5wgqc" event={"ID":"b0947dc1-afc5-4358-8f25-622c94493529","Type":"ContainerDied","Data":"8845f22801240dff0dc65955641dbd9a535976689586a65b33173bc035682d0c"} Sep 09 10:55:58.631271 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:58.631236 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e5f9c5-rbmcj"] Sep 09 10:55:58.632835 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:58.632791 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e5f9c5-rbmcj"] Sep 09 10:55:58.737951 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:58.737924 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e5f9c5-64b65fcc45-jg2l7" Sep 09 10:55:58.741354 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:58.741330 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e5f9c5-8985b6d54-5wgqc" Sep 09 10:55:58.817187 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:58.817150 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a7a18d-8aab-4b7f-8bdb-68bec973a43a-plugins-config-volume\") pod \"b9a7a18d-8aab-4b7f-8bdb-68bec973a43a\" (UID: \"b9a7a18d-8aab-4b7f-8bdb-68bec973a43a\") " Sep 09 10:55:58.817365 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:58.817199 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0947dc1-afc5-4358-8f25-622c94493529-plugins-config-volume\") pod \"b0947dc1-afc5-4358-8f25-622c94493529\" (UID: \"b0947dc1-afc5-4358-8f25-622c94493529\") " Sep 09 10:55:58.817365 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:58.817291 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nj9ms\" (UniqueName: \"kubernetes.io/projected/b9a7a18d-8aab-4b7f-8bdb-68bec973a43a-kube-api-access-nj9ms\") pod \"b9a7a18d-8aab-4b7f-8bdb-68bec973a43a\" (UID: \"b9a7a18d-8aab-4b7f-8bdb-68bec973a43a\") " Sep 09 10:55:58.817365 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:58.817319 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ttlt\" (UniqueName: \"kubernetes.io/projected/b0947dc1-afc5-4358-8f25-622c94493529-kube-api-access-9ttlt\") pod \"b0947dc1-afc5-4358-8f25-622c94493529\" (UID: \"b0947dc1-afc5-4358-8f25-622c94493529\") " Sep 09 10:55:58.817714 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:58.817675 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a7a18d-8aab-4b7f-8bdb-68bec973a43a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b9a7a18d-8aab-4b7f-8bdb-68bec973a43a" (UID: "b9a7a18d-8aab-4b7f-8bdb-68bec973a43a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:55:58.817714 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:58.817685 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0947dc1-afc5-4358-8f25-622c94493529-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b0947dc1-afc5-4358-8f25-622c94493529" (UID: "b0947dc1-afc5-4358-8f25-622c94493529"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:55:58.819560 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:58.819534 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0947dc1-afc5-4358-8f25-622c94493529-kube-api-access-9ttlt" (OuterVolumeSpecName: "kube-api-access-9ttlt") pod "b0947dc1-afc5-4358-8f25-622c94493529" (UID: "b0947dc1-afc5-4358-8f25-622c94493529"). InnerVolumeSpecName "kube-api-access-9ttlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:55:58.819560 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:58.819546 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a7a18d-8aab-4b7f-8bdb-68bec973a43a-kube-api-access-nj9ms" (OuterVolumeSpecName: "kube-api-access-nj9ms") pod "b9a7a18d-8aab-4b7f-8bdb-68bec973a43a" (UID: "b9a7a18d-8aab-4b7f-8bdb-68bec973a43a"). InnerVolumeSpecName "kube-api-access-nj9ms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:55:58.918993 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:58.918948 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a7a18d-8aab-4b7f-8bdb-68bec973a43a-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:55:58.918993 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:58.918983 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0947dc1-afc5-4358-8f25-622c94493529-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:55:58.918993 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:58.918992 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj9ms\" (UniqueName: \"kubernetes.io/projected/b9a7a18d-8aab-4b7f-8bdb-68bec973a43a-kube-api-access-nj9ms\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:55:58.918993 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:58.919002 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ttlt\" (UniqueName: \"kubernetes.io/projected/b0947dc1-afc5-4358-8f25-622c94493529-kube-api-access-9ttlt\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:55:59.568207 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:59.568179 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e5f9c5-64b65fcc45-jg2l7" Sep 09 10:55:59.568760 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:59.568175 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e5f9c5-64b65fcc45-jg2l7" event={"ID":"b9a7a18d-8aab-4b7f-8bdb-68bec973a43a","Type":"ContainerDied","Data":"553a478e906a4112adc6c4a63e165df2ad20eeb9db3ee0c3eb9c94d5ce364bab"} Sep 09 10:55:59.568760 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:59.568304 2559 scope.go:117] "RemoveContainer" containerID="2d64e59b2dcd504504e93bb2c7c590e4a5be3c3c1af5157e468316b809aede43" Sep 09 10:55:59.569728 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:59.569671 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e5f9c5-8985b6d54-5wgqc" Sep 09 10:55:59.569728 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:59.569712 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e5f9c5-8985b6d54-5wgqc" event={"ID":"b0947dc1-afc5-4358-8f25-622c94493529","Type":"ContainerDied","Data":"f96255630480102a480580e64f87830107febfec06bb56ec2628b104b90998df"} Sep 09 10:55:59.579838 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:59.579817 2559 scope.go:117] "RemoveContainer" containerID="8845f22801240dff0dc65955641dbd9a535976689586a65b33173bc035682d0c" Sep 09 10:55:59.595483 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:59.595451 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e5f9c5-8985b6d54-5wgqc"] Sep 09 10:55:59.601573 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:59.601547 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e5f9c5-8985b6d54-5wgqc"] Sep 09 10:55:59.613793 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:59.613762 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e5f9c5-64b65fcc45-jg2l7"] Sep 09 10:55:59.619116 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:55:59.619088 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e5f9c5-64b65fcc45-jg2l7"] Sep 09 10:56:00.334981 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:00.334947 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0947dc1-afc5-4358-8f25-622c94493529" path="/var/lib/kubelet/pods/b0947dc1-afc5-4358-8f25-622c94493529/volumes" Sep 09 10:56:00.335311 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:00.335298 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9a7a18d-8aab-4b7f-8bdb-68bec973a43a" path="/var/lib/kubelet/pods/b9a7a18d-8aab-4b7f-8bdb-68bec973a43a/volumes" Sep 09 10:56:00.335608 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:00.335595 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dab98061-a35b-4a6e-816d-3e0262d66767" path="/var/lib/kubelet/pods/dab98061-a35b-4a6e-816d-3e0262d66767/volumes" Sep 09 10:56:00.380855 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:00.380827 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d22ff_0a56b285-3f3c-4f6d-a939-8a95ae00e8da/ovn-acl-logging/0.log" Sep 09 10:56:00.385804 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:00.385781 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d22ff_0a56b285-3f3c-4f6d-a939-8a95ae00e8da/ovn-acl-logging/0.log" Sep 09 10:56:00.584300 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:00.584264 2559 scope.go:117] "RemoveContainer" containerID="e2e548f8fdc85d6bd5b10c78470dac61eab5b0e96747ce8b09a1816c766d206f" Sep 09 10:56:05.802124 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:05.802085 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-798524-krd7p"] Sep 09 10:56:05.802555 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:05.802473 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0947dc1-afc5-4358-8f25-622c94493529" containerName="payload-pre-processing" Sep 09 10:56:05.802555 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:05.802484 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0947dc1-afc5-4358-8f25-622c94493529" containerName="payload-pre-processing" Sep 09 10:56:05.802555 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:05.802522 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dab98061-a35b-4a6e-816d-3e0262d66767" containerName="revoke-keys" Sep 09 10:56:05.802555 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:05.802532 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dab98061-a35b-4a6e-816d-3e0262d66767" containerName="revoke-keys" Sep 09 10:56:05.802555 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:05.802551 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9a7a18d-8aab-4b7f-8bdb-68bec973a43a" containerName="payload-processing" Sep 09 10:56:05.802555 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:05.802556 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9a7a18d-8aab-4b7f-8bdb-68bec973a43a" containerName="payload-processing" Sep 09 10:56:05.802761 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:05.802616 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="dab98061-a35b-4a6e-816d-3e0262d66767" containerName="revoke-keys" Sep 09 10:56:05.802761 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:05.802626 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0947dc1-afc5-4358-8f25-622c94493529" containerName="payload-pre-processing" Sep 09 10:56:05.802761 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:05.802635 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9a7a18d-8aab-4b7f-8bdb-68bec973a43a" containerName="payload-processing" Sep 09 10:56:05.805748 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:05.805730 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-798524-krd7p" Sep 09 10:56:05.808333 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:05.808311 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 10:56:05.808413 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:05.808371 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kq7qv\"" Sep 09 10:56:05.813184 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:05.813156 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-798524-krd7p"] Sep 09 10:56:05.880979 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:05.880942 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eca33b6-3147-46d0-9bed-dc8b64f9d61f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-798524-krd7p\" (UID: \"5eca33b6-3147-46d0-9bed-dc8b64f9d61f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-798524-krd7p" Sep 09 10:56:05.981776 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:05.981743 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eca33b6-3147-46d0-9bed-dc8b64f9d61f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-798524-krd7p\" (UID: \"5eca33b6-3147-46d0-9bed-dc8b64f9d61f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-798524-krd7p" Sep 09 10:56:05.982383 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:05.982356 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eca33b6-3147-46d0-9bed-dc8b64f9d61f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-798524-krd7p\" (UID: \"5eca33b6-3147-46d0-9bed-dc8b64f9d61f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-798524-krd7p" Sep 09 10:56:06.118617 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:06.118574 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-798524-krd7p" Sep 09 10:56:06.392139 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:06.392113 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-798524-krd7p"] Sep 09 10:56:06.394736 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:56:06.394707 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eca33b6_3147_46d0_9bed_dc8b64f9d61f.slice/crio-c639da6bb37b2336e08933df404175a2e34d54c496afc7b2b3998014e2eac9ae WatchSource:0}: Error finding container c639da6bb37b2336e08933df404175a2e34d54c496afc7b2b3998014e2eac9ae: Status 404 returned error can't find the container with id c639da6bb37b2336e08933df404175a2e34d54c496afc7b2b3998014e2eac9ae Sep 09 10:56:06.608760 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:06.608722 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-798524-krd7p" event={"ID":"5eca33b6-3147-46d0-9bed-dc8b64f9d61f","Type":"ContainerStarted","Data":"c639da6bb37b2336e08933df404175a2e34d54c496afc7b2b3998014e2eac9ae"} Sep 09 10:56:07.615860 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:07.615822 2559 generic.go:356] "Generic (PLEG): container finished" podID="5eca33b6-3147-46d0-9bed-dc8b64f9d61f" containerID="7f758de8ba9335a5b7d7f1a7a5d28b7c4604cd77cea002fb9dedf1608b422e32" exitCode=0 Sep 09 10:56:07.616259 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:07.615888 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-798524-krd7p" event={"ID":"5eca33b6-3147-46d0-9bed-dc8b64f9d61f","Type":"ContainerDied","Data":"7f758de8ba9335a5b7d7f1a7a5d28b7c4604cd77cea002fb9dedf1608b422e32"} Sep 09 10:56:08.759085 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:08.759057 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-798524-krd7p" Sep 09 10:56:08.809488 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:08.809450 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eca33b6-3147-46d0-9bed-dc8b64f9d61f-maas-api-service-ca\") pod \"5eca33b6-3147-46d0-9bed-dc8b64f9d61f\" (UID: \"5eca33b6-3147-46d0-9bed-dc8b64f9d61f\") " Sep 09 10:56:08.809867 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:08.809831 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eca33b6-3147-46d0-9bed-dc8b64f9d61f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5eca33b6-3147-46d0-9bed-dc8b64f9d61f" (UID: "5eca33b6-3147-46d0-9bed-dc8b64f9d61f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:56:08.910632 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:08.910545 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eca33b6-3147-46d0-9bed-dc8b64f9d61f-maas-api-service-ca\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:56:09.627738 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:09.627708 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-798524-krd7p" Sep 09 10:56:09.627931 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:09.627707 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-798524-krd7p" event={"ID":"5eca33b6-3147-46d0-9bed-dc8b64f9d61f","Type":"ContainerDied","Data":"c639da6bb37b2336e08933df404175a2e34d54c496afc7b2b3998014e2eac9ae"} Sep 09 10:56:09.627931 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:09.627874 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c639da6bb37b2336e08933df404175a2e34d54c496afc7b2b3998014e2eac9ae" Sep 09 10:56:21.449740 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:21.449053 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-798524-5bcf6b74c-k7frc"] Sep 09 10:56:21.449740 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:21.449290 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-798524-5bcf6b74c-k7frc" podUID="5afa4a78-e5b7-4108-8dba-b1a36fa133d6" containerName="payload-processing" containerID="cri-o://bb49fe3bd29b634aae76bfc23fa6a3b78d0ad7e2c53c33b2751491534f43dc0d" gracePeriod=30 Sep 09 10:56:21.658051 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:21.658009 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-798524-krd7p"] Sep 09 10:56:21.660836 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:21.660807 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-798524-krd7p"] Sep 09 10:56:21.695570 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:21.695536 2559 generic.go:356] "Generic (PLEG): container finished" podID="5afa4a78-e5b7-4108-8dba-b1a36fa133d6" containerID="bb49fe3bd29b634aae76bfc23fa6a3b78d0ad7e2c53c33b2751491534f43dc0d" exitCode=0 Sep 09 10:56:21.695753 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:21.695603 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-798524-5bcf6b74c-k7frc" event={"ID":"5afa4a78-e5b7-4108-8dba-b1a36fa133d6","Type":"ContainerDied","Data":"bb49fe3bd29b634aae76bfc23fa6a3b78d0ad7e2c53c33b2751491534f43dc0d"} Sep 09 10:56:21.726300 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:21.726265 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-798524-5bcf6b74c-k7frc" Sep 09 10:56:21.827412 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:21.827366 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5afa4a78-e5b7-4108-8dba-b1a36fa133d6-plugins-config-volume\") pod \"5afa4a78-e5b7-4108-8dba-b1a36fa133d6\" (UID: \"5afa4a78-e5b7-4108-8dba-b1a36fa133d6\") " Sep 09 10:56:21.827658 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:21.827472 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-df6mb\" (UniqueName: \"kubernetes.io/projected/5afa4a78-e5b7-4108-8dba-b1a36fa133d6-kube-api-access-df6mb\") pod \"5afa4a78-e5b7-4108-8dba-b1a36fa133d6\" (UID: \"5afa4a78-e5b7-4108-8dba-b1a36fa133d6\") " Sep 09 10:56:21.827894 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:21.827865 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5afa4a78-e5b7-4108-8dba-b1a36fa133d6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5afa4a78-e5b7-4108-8dba-b1a36fa133d6" (UID: "5afa4a78-e5b7-4108-8dba-b1a36fa133d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:56:21.829721 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:21.829691 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5afa4a78-e5b7-4108-8dba-b1a36fa133d6-kube-api-access-df6mb" (OuterVolumeSpecName: "kube-api-access-df6mb") pod "5afa4a78-e5b7-4108-8dba-b1a36fa133d6" (UID: "5afa4a78-e5b7-4108-8dba-b1a36fa133d6"). InnerVolumeSpecName "kube-api-access-df6mb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:56:21.928914 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:21.928869 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5afa4a78-e5b7-4108-8dba-b1a36fa133d6-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:56:21.928914 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:21.928904 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df6mb\" (UniqueName: \"kubernetes.io/projected/5afa4a78-e5b7-4108-8dba-b1a36fa133d6-kube-api-access-df6mb\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:56:22.334969 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:22.334933 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5eca33b6-3147-46d0-9bed-dc8b64f9d61f" path="/var/lib/kubelet/pods/5eca33b6-3147-46d0-9bed-dc8b64f9d61f/volumes" Sep 09 10:56:22.702223 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:22.702142 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-798524-5bcf6b74c-k7frc" Sep 09 10:56:22.702223 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:22.702147 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-798524-5bcf6b74c-k7frc" event={"ID":"5afa4a78-e5b7-4108-8dba-b1a36fa133d6","Type":"ContainerDied","Data":"bda5690702a062de19e5c1beeb87d8d7fe8f7bff01c5b94fa74cbbb65fd67aaa"} Sep 09 10:56:22.702223 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:22.702203 2559 scope.go:117] "RemoveContainer" containerID="bb49fe3bd29b634aae76bfc23fa6a3b78d0ad7e2c53c33b2751491534f43dc0d" Sep 09 10:56:22.718897 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:22.718867 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-798524-5bcf6b74c-k7frc"] Sep 09 10:56:22.723469 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:22.723440 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-798524-5bcf6b74c-k7frc"] Sep 09 10:56:24.334131 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:24.334091 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5afa4a78-e5b7-4108-8dba-b1a36fa133d6" path="/var/lib/kubelet/pods/5afa4a78-e5b7-4108-8dba-b1a36fa133d6/volumes" Sep 09 10:56:29.249164 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:29.249122 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-62f893-hlmcp"] Sep 09 10:56:29.251394 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:29.249569 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5afa4a78-e5b7-4108-8dba-b1a36fa133d6" containerName="payload-processing" Sep 09 10:56:29.251394 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:29.249583 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5afa4a78-e5b7-4108-8dba-b1a36fa133d6" containerName="payload-processing" Sep 09 10:56:29.251394 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:29.249598 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eca33b6-3147-46d0-9bed-dc8b64f9d61f" containerName="revoke-keys" Sep 09 10:56:29.251394 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:29.249604 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eca33b6-3147-46d0-9bed-dc8b64f9d61f" containerName="revoke-keys" Sep 09 10:56:29.251394 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:29.249676 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5eca33b6-3147-46d0-9bed-dc8b64f9d61f" containerName="revoke-keys" Sep 09 10:56:29.251394 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:29.249686 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5afa4a78-e5b7-4108-8dba-b1a36fa133d6" containerName="payload-processing" Sep 09 10:56:29.252713 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:29.252693 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-62f893-hlmcp" Sep 09 10:56:29.256205 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:29.256179 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 10:56:29.256360 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:29.256259 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kq7qv\"" Sep 09 10:56:29.261988 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:29.261942 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-62f893-hlmcp"] Sep 09 10:56:29.393806 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:29.393764 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6328347-f024-4281-9908-98f4e5be9c2a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-62f893-hlmcp\" (UID: \"a6328347-f024-4281-9908-98f4e5be9c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-62f893-hlmcp" Sep 09 10:56:29.494800 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:29.494756 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6328347-f024-4281-9908-98f4e5be9c2a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-62f893-hlmcp\" (UID: \"a6328347-f024-4281-9908-98f4e5be9c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-62f893-hlmcp" Sep 09 10:56:29.495467 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:29.495448 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6328347-f024-4281-9908-98f4e5be9c2a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-62f893-hlmcp\" (UID: \"a6328347-f024-4281-9908-98f4e5be9c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-62f893-hlmcp" Sep 09 10:56:29.565849 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:29.565755 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-62f893-hlmcp" Sep 09 10:56:29.847867 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:29.847839 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-62f893-hlmcp"] Sep 09 10:56:29.850304 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:56:29.850273 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6328347_f024_4281_9908_98f4e5be9c2a.slice/crio-7b52773a4c3253e4704770de0bf26547d8780b1366263ad6e4f6b6871926ba41 WatchSource:0}: Error finding container 7b52773a4c3253e4704770de0bf26547d8780b1366263ad6e4f6b6871926ba41: Status 404 returned error can't find the container with id 7b52773a4c3253e4704770de0bf26547d8780b1366263ad6e4f6b6871926ba41 Sep 09 10:56:30.747051 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:30.747002 2559 generic.go:356] "Generic (PLEG): container finished" podID="a6328347-f024-4281-9908-98f4e5be9c2a" containerID="eb00429de281b049701c9a2631c6e5a4da5e2624077350872b7360cc9decaffe" exitCode=0 Sep 09 10:56:30.747051 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:30.747040 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-62f893-hlmcp" event={"ID":"a6328347-f024-4281-9908-98f4e5be9c2a","Type":"ContainerDied","Data":"eb00429de281b049701c9a2631c6e5a4da5e2624077350872b7360cc9decaffe"} Sep 09 10:56:30.747705 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:30.747113 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-62f893-hlmcp" event={"ID":"a6328347-f024-4281-9908-98f4e5be9c2a","Type":"ContainerStarted","Data":"7b52773a4c3253e4704770de0bf26547d8780b1366263ad6e4f6b6871926ba41"} Sep 09 10:56:31.895774 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:31.895741 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-62f893-hlmcp" Sep 09 10:56:32.017863 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:32.017767 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6328347-f024-4281-9908-98f4e5be9c2a-maas-api-service-ca\") pod \"a6328347-f024-4281-9908-98f4e5be9c2a\" (UID: \"a6328347-f024-4281-9908-98f4e5be9c2a\") " Sep 09 10:56:32.018163 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:32.018138 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6328347-f024-4281-9908-98f4e5be9c2a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a6328347-f024-4281-9908-98f4e5be9c2a" (UID: "a6328347-f024-4281-9908-98f4e5be9c2a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:56:32.118931 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:32.118888 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6328347-f024-4281-9908-98f4e5be9c2a-maas-api-service-ca\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:56:32.759090 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:32.759000 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-62f893-hlmcp" Sep 09 10:56:32.759090 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:32.759034 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-62f893-hlmcp" event={"ID":"a6328347-f024-4281-9908-98f4e5be9c2a","Type":"ContainerDied","Data":"7b52773a4c3253e4704770de0bf26547d8780b1366263ad6e4f6b6871926ba41"} Sep 09 10:56:32.759090 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:32.759080 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b52773a4c3253e4704770de0bf26547d8780b1366263ad6e4f6b6871926ba41" Sep 09 10:56:36.007956 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:36.007917 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2"] Sep 09 10:56:39.989640 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:39.989604 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-62f893-6757df5f6b-vt8nx"] Sep 09 10:56:39.990050 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:39.989865 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-62f893-6757df5f6b-vt8nx" podUID="5d14c4c8-3a61-48b6-a94d-55b5edd08b46" containerName="payload-pre-processing" containerID="cri-o://f254ac4b72db040f828a087b14ce6d6c79ef3769640f05eb9bcb4cd2faa4a33b" gracePeriod=30 Sep 09 10:56:40.249356 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:40.249299 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-62f893-6757df5f6b-vt8nx" Sep 09 10:56:40.290178 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:40.290137 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-62f893-hlmcp"] Sep 09 10:56:40.292486 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:40.292463 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-62f893-hlmcp"] Sep 09 10:56:40.335541 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:40.335495 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6328347-f024-4281-9908-98f4e5be9c2a" path="/var/lib/kubelet/pods/a6328347-f024-4281-9908-98f4e5be9c2a/volumes" Sep 09 10:56:40.396807 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:40.396773 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d14c4c8-3a61-48b6-a94d-55b5edd08b46-plugins-config-volume\") pod \"5d14c4c8-3a61-48b6-a94d-55b5edd08b46\" (UID: \"5d14c4c8-3a61-48b6-a94d-55b5edd08b46\") " Sep 09 10:56:40.396994 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:40.396817 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tbs9\" (UniqueName: \"kubernetes.io/projected/5d14c4c8-3a61-48b6-a94d-55b5edd08b46-kube-api-access-5tbs9\") pod \"5d14c4c8-3a61-48b6-a94d-55b5edd08b46\" (UID: \"5d14c4c8-3a61-48b6-a94d-55b5edd08b46\") " Sep 09 10:56:40.397204 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:40.397179 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d14c4c8-3a61-48b6-a94d-55b5edd08b46-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5d14c4c8-3a61-48b6-a94d-55b5edd08b46" (UID: "5d14c4c8-3a61-48b6-a94d-55b5edd08b46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:56:40.398982 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:40.398953 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d14c4c8-3a61-48b6-a94d-55b5edd08b46-kube-api-access-5tbs9" (OuterVolumeSpecName: "kube-api-access-5tbs9") pod "5d14c4c8-3a61-48b6-a94d-55b5edd08b46" (UID: "5d14c4c8-3a61-48b6-a94d-55b5edd08b46"). InnerVolumeSpecName "kube-api-access-5tbs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:56:40.497339 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:40.497306 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d14c4c8-3a61-48b6-a94d-55b5edd08b46-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:56:40.497339 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:40.497335 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tbs9\" (UniqueName: \"kubernetes.io/projected/5d14c4c8-3a61-48b6-a94d-55b5edd08b46-kube-api-access-5tbs9\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:56:40.798923 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:40.798889 2559 generic.go:356] "Generic (PLEG): container finished" podID="5d14c4c8-3a61-48b6-a94d-55b5edd08b46" containerID="f254ac4b72db040f828a087b14ce6d6c79ef3769640f05eb9bcb4cd2faa4a33b" exitCode=0 Sep 09 10:56:40.799090 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:40.798931 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-62f893-6757df5f6b-vt8nx" Sep 09 10:56:40.799090 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:40.798976 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-62f893-6757df5f6b-vt8nx" event={"ID":"5d14c4c8-3a61-48b6-a94d-55b5edd08b46","Type":"ContainerDied","Data":"f254ac4b72db040f828a087b14ce6d6c79ef3769640f05eb9bcb4cd2faa4a33b"} Sep 09 10:56:40.799090 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:40.799027 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-62f893-6757df5f6b-vt8nx" event={"ID":"5d14c4c8-3a61-48b6-a94d-55b5edd08b46","Type":"ContainerDied","Data":"507922e20450a574f155cedcbad248064aba2be1e31b05d0be485ca088dddb44"} Sep 09 10:56:40.799090 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:40.799051 2559 scope.go:117] "RemoveContainer" containerID="f254ac4b72db040f828a087b14ce6d6c79ef3769640f05eb9bcb4cd2faa4a33b" Sep 09 10:56:40.808200 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:40.808183 2559 scope.go:117] "RemoveContainer" containerID="f254ac4b72db040f828a087b14ce6d6c79ef3769640f05eb9bcb4cd2faa4a33b" Sep 09 10:56:40.808473 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:56:40.808457 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f254ac4b72db040f828a087b14ce6d6c79ef3769640f05eb9bcb4cd2faa4a33b\": container with ID starting with f254ac4b72db040f828a087b14ce6d6c79ef3769640f05eb9bcb4cd2faa4a33b not found: ID does not exist" containerID="f254ac4b72db040f828a087b14ce6d6c79ef3769640f05eb9bcb4cd2faa4a33b" Sep 09 10:56:40.808555 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:40.808479 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f254ac4b72db040f828a087b14ce6d6c79ef3769640f05eb9bcb4cd2faa4a33b"} err="failed to get container status \"f254ac4b72db040f828a087b14ce6d6c79ef3769640f05eb9bcb4cd2faa4a33b\": rpc error: code = NotFound desc = could not find container \"f254ac4b72db040f828a087b14ce6d6c79ef3769640f05eb9bcb4cd2faa4a33b\": container with ID starting with f254ac4b72db040f828a087b14ce6d6c79ef3769640f05eb9bcb4cd2faa4a33b not found: ID does not exist" Sep 09 10:56:40.825587 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:40.825560 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-62f893-6757df5f6b-vt8nx"] Sep 09 10:56:40.827470 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:40.827446 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-62f893-6757df5f6b-vt8nx"] Sep 09 10:56:42.334570 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:42.334535 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d14c4c8-3a61-48b6-a94d-55b5edd08b46" path="/var/lib/kubelet/pods/5d14c4c8-3a61-48b6-a94d-55b5edd08b46/volumes" Sep 09 10:56:43.183046 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:43.183009 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-33d685-7c7cd7c479-rbhzn"] Sep 09 10:56:43.183424 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:43.183410 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d14c4c8-3a61-48b6-a94d-55b5edd08b46" containerName="payload-pre-processing" Sep 09 10:56:43.183476 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:43.183425 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d14c4c8-3a61-48b6-a94d-55b5edd08b46" containerName="payload-pre-processing" Sep 09 10:56:43.183476 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:43.183445 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6328347-f024-4281-9908-98f4e5be9c2a" containerName="revoke-keys" Sep 09 10:56:43.183476 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:43.183451 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6328347-f024-4281-9908-98f4e5be9c2a" containerName="revoke-keys" Sep 09 10:56:43.183604 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:43.183522 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d14c4c8-3a61-48b6-a94d-55b5edd08b46" containerName="payload-pre-processing" Sep 09 10:56:43.183604 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:43.183534 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6328347-f024-4281-9908-98f4e5be9c2a" containerName="revoke-keys" Sep 09 10:56:43.188090 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:43.188066 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-33d685-7c7cd7c479-rbhzn" Sep 09 10:56:43.190964 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:43.190945 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-33d685-dockercfg-hln7r\"" Sep 09 10:56:43.191076 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:43.190984 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-33d685\"" Sep 09 10:56:43.199228 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:43.199200 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-33d685-7c7cd7c479-rbhzn"] Sep 09 10:56:43.216946 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:43.216908 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h54lf\" (UniqueName: \"kubernetes.io/projected/60aea43a-6bf8-406d-80e2-61b46e25c98f-kube-api-access-h54lf\") pod \"payload-processing-e2e-ipp-a-33d685-7c7cd7c479-rbhzn\" (UID: \"60aea43a-6bf8-406d-80e2-61b46e25c98f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-33d685-7c7cd7c479-rbhzn" Sep 09 10:56:43.216946 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:43.216955 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60aea43a-6bf8-406d-80e2-61b46e25c98f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-33d685-7c7cd7c479-rbhzn\" (UID: \"60aea43a-6bf8-406d-80e2-61b46e25c98f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-33d685-7c7cd7c479-rbhzn" Sep 09 10:56:43.317658 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:43.317624 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h54lf\" (UniqueName: \"kubernetes.io/projected/60aea43a-6bf8-406d-80e2-61b46e25c98f-kube-api-access-h54lf\") pod \"payload-processing-e2e-ipp-a-33d685-7c7cd7c479-rbhzn\" (UID: \"60aea43a-6bf8-406d-80e2-61b46e25c98f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-33d685-7c7cd7c479-rbhzn" Sep 09 10:56:43.317859 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:43.317681 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60aea43a-6bf8-406d-80e2-61b46e25c98f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-33d685-7c7cd7c479-rbhzn\" (UID: \"60aea43a-6bf8-406d-80e2-61b46e25c98f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-33d685-7c7cd7c479-rbhzn" Sep 09 10:56:43.318560 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:43.318537 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60aea43a-6bf8-406d-80e2-61b46e25c98f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-33d685-7c7cd7c479-rbhzn\" (UID: \"60aea43a-6bf8-406d-80e2-61b46e25c98f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-33d685-7c7cd7c479-rbhzn" Sep 09 10:56:43.327226 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:43.327194 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h54lf\" (UniqueName: \"kubernetes.io/projected/60aea43a-6bf8-406d-80e2-61b46e25c98f-kube-api-access-h54lf\") pod \"payload-processing-e2e-ipp-a-33d685-7c7cd7c479-rbhzn\" (UID: \"60aea43a-6bf8-406d-80e2-61b46e25c98f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-33d685-7c7cd7c479-rbhzn" Sep 09 10:56:43.499442 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:43.499351 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-33d685-7c7cd7c479-rbhzn" Sep 09 10:56:43.777111 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:43.777087 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-33d685-7c7cd7c479-rbhzn"] Sep 09 10:56:43.779287 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:56:43.779262 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60aea43a_6bf8_406d_80e2_61b46e25c98f.slice/crio-a9a0073176513f91b0fd932950857f8c86341e815a56265cff004632bc3ca2aa WatchSource:0}: Error finding container a9a0073176513f91b0fd932950857f8c86341e815a56265cff004632bc3ca2aa: Status 404 returned error can't find the container with id a9a0073176513f91b0fd932950857f8c86341e815a56265cff004632bc3ca2aa Sep 09 10:56:43.818541 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:43.818481 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-33d685-7c7cd7c479-rbhzn" event={"ID":"60aea43a-6bf8-406d-80e2-61b46e25c98f","Type":"ContainerStarted","Data":"a9a0073176513f91b0fd932950857f8c86341e815a56265cff004632bc3ca2aa"} Sep 09 10:56:44.825406 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:44.825370 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-33d685-7c7cd7c479-rbhzn" event={"ID":"60aea43a-6bf8-406d-80e2-61b46e25c98f","Type":"ContainerStarted","Data":"705fb2eff2472b74bf77b4276cf8e06250045056f426c094708763699f60fd81"} Sep 09 10:56:44.825812 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:44.825438 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-33d685-7c7cd7c479-rbhzn" Sep 09 10:56:47.686395 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:47.686332 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-33d685-7c7cd7c479-rbhzn" podStartSLOduration=4.686311672 podStartE2EDuration="4.686311672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:56:44.849457337 +0000 UTC m=+1845.024059676" watchObservedRunningTime="2026-09-09 10:56:47.686311672 +0000 UTC m=+1847.860913997" Sep 09 10:56:47.687657 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:47.687634 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a03d9e-582jg"] Sep 09 10:56:47.691220 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:47.691204 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a03d9e-582jg" Sep 09 10:56:47.693977 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:47.693955 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 10:56:47.694081 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:47.693981 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kq7qv\"" Sep 09 10:56:47.699224 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:47.699202 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a03d9e-582jg"] Sep 09 10:56:47.750402 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:47.750359 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bc3071f-e672-42cf-a1b5-15447cef32ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a03d9e-582jg\" (UID: \"8bc3071f-e672-42cf-a1b5-15447cef32ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a03d9e-582jg" Sep 09 10:56:47.850918 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:47.850880 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bc3071f-e672-42cf-a1b5-15447cef32ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a03d9e-582jg\" (UID: \"8bc3071f-e672-42cf-a1b5-15447cef32ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a03d9e-582jg" Sep 09 10:56:47.851573 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:47.851549 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bc3071f-e672-42cf-a1b5-15447cef32ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a03d9e-582jg\" (UID: \"8bc3071f-e672-42cf-a1b5-15447cef32ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a03d9e-582jg" Sep 09 10:56:48.002354 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:48.002267 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a03d9e-582jg" Sep 09 10:56:48.284880 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:48.284844 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a03d9e-582jg"] Sep 09 10:56:48.287539 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:56:48.287494 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc3071f_e672_42cf_a1b5_15447cef32ad.slice/crio-99201b66c7b62b1dee3ed3ce1003a6d23262c87b66000431ecb277f02b2da022 WatchSource:0}: Error finding container 99201b66c7b62b1dee3ed3ce1003a6d23262c87b66000431ecb277f02b2da022: Status 404 returned error can't find the container with id 99201b66c7b62b1dee3ed3ce1003a6d23262c87b66000431ecb277f02b2da022 Sep 09 10:56:48.848486 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:48.848453 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a03d9e-582jg" event={"ID":"8bc3071f-e672-42cf-a1b5-15447cef32ad","Type":"ContainerStarted","Data":"60069030d7540c7fcc8930d9661fb24512fba583ed0c687cca2e49dd6290f601"} Sep 09 10:56:48.848901 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:48.848535 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a03d9e-582jg" event={"ID":"8bc3071f-e672-42cf-a1b5-15447cef32ad","Type":"ContainerStarted","Data":"99201b66c7b62b1dee3ed3ce1003a6d23262c87b66000431ecb277f02b2da022"} Sep 09 10:56:48.867806 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:48.867758 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a03d9e-582jg" podStartSLOduration=1.867740998 podStartE2EDuration="1.867740998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:56:48.866047769 +0000 UTC m=+1849.040650097" watchObservedRunningTime="2026-09-09 10:56:48.867740998 +0000 UTC m=+1849.042343322" Sep 09 10:56:49.855832 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:49.855789 2559 generic.go:356] "Generic (PLEG): container finished" podID="8bc3071f-e672-42cf-a1b5-15447cef32ad" containerID="60069030d7540c7fcc8930d9661fb24512fba583ed0c687cca2e49dd6290f601" exitCode=0 Sep 09 10:56:49.856238 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:49.855874 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a03d9e-582jg" event={"ID":"8bc3071f-e672-42cf-a1b5-15447cef32ad","Type":"ContainerDied","Data":"60069030d7540c7fcc8930d9661fb24512fba583ed0c687cca2e49dd6290f601"} Sep 09 10:56:51.000915 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.000891 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a03d9e-582jg" Sep 09 10:56:51.027815 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.027773 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" podUID="02cd273e-83d2-4c79-a636-ab2b33e46117" containerName="main" containerID="cri-o://e3cf662d74c75475c27046f2785b8a6d00c8089c2ccf54ad9c87bc921b4bdd1d" gracePeriod=45 Sep 09 10:56:51.074433 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.074395 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bc3071f-e672-42cf-a1b5-15447cef32ad-maas-api-service-ca\") pod \"8bc3071f-e672-42cf-a1b5-15447cef32ad\" (UID: \"8bc3071f-e672-42cf-a1b5-15447cef32ad\") " Sep 09 10:56:51.074812 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.074785 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bc3071f-e672-42cf-a1b5-15447cef32ad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8bc3071f-e672-42cf-a1b5-15447cef32ad" (UID: "8bc3071f-e672-42cf-a1b5-15447cef32ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:56:51.175458 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.175423 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bc3071f-e672-42cf-a1b5-15447cef32ad-maas-api-service-ca\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:56:51.287941 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.287913 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" Sep 09 10:56:51.378282 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.378238 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02cd273e-83d2-4c79-a636-ab2b33e46117-model-cache\") pod \"02cd273e-83d2-4c79-a636-ab2b33e46117\" (UID: \"02cd273e-83d2-4c79-a636-ab2b33e46117\") " Sep 09 10:56:51.378282 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.378289 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02cd273e-83d2-4c79-a636-ab2b33e46117-home\") pod \"02cd273e-83d2-4c79-a636-ab2b33e46117\" (UID: \"02cd273e-83d2-4c79-a636-ab2b33e46117\") " Sep 09 10:56:51.378593 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.378331 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dg8j8\" (UniqueName: \"kubernetes.io/projected/02cd273e-83d2-4c79-a636-ab2b33e46117-kube-api-access-dg8j8\") pod \"02cd273e-83d2-4c79-a636-ab2b33e46117\" (UID: \"02cd273e-83d2-4c79-a636-ab2b33e46117\") " Sep 09 10:56:51.378593 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.378363 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02cd273e-83d2-4c79-a636-ab2b33e46117-dshm\") pod \"02cd273e-83d2-4c79-a636-ab2b33e46117\" (UID: \"02cd273e-83d2-4c79-a636-ab2b33e46117\") " Sep 09 10:56:51.378593 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.378425 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02cd273e-83d2-4c79-a636-ab2b33e46117-tmp-dir\") pod \"02cd273e-83d2-4c79-a636-ab2b33e46117\" (UID: \"02cd273e-83d2-4c79-a636-ab2b33e46117\") " Sep 09 10:56:51.378593 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.378446 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02cd273e-83d2-4c79-a636-ab2b33e46117-tls-certs\") pod \"02cd273e-83d2-4c79-a636-ab2b33e46117\" (UID: \"02cd273e-83d2-4c79-a636-ab2b33e46117\") " Sep 09 10:56:51.378795 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.378619 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02cd273e-83d2-4c79-a636-ab2b33e46117-model-cache" (OuterVolumeSpecName: "model-cache") pod "02cd273e-83d2-4c79-a636-ab2b33e46117" (UID: "02cd273e-83d2-4c79-a636-ab2b33e46117"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 10:56:51.378795 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.378627 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02cd273e-83d2-4c79-a636-ab2b33e46117-home" (OuterVolumeSpecName: "home") pod "02cd273e-83d2-4c79-a636-ab2b33e46117" (UID: "02cd273e-83d2-4c79-a636-ab2b33e46117"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 10:56:51.378795 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.378774 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02cd273e-83d2-4c79-a636-ab2b33e46117-model-cache\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:56:51.378795 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.378768 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02cd273e-83d2-4c79-a636-ab2b33e46117-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "02cd273e-83d2-4c79-a636-ab2b33e46117" (UID: "02cd273e-83d2-4c79-a636-ab2b33e46117"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 10:56:51.378795 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.378793 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02cd273e-83d2-4c79-a636-ab2b33e46117-home\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:56:51.380572 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.380547 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02cd273e-83d2-4c79-a636-ab2b33e46117-dshm" (OuterVolumeSpecName: "dshm") pod "02cd273e-83d2-4c79-a636-ab2b33e46117" (UID: "02cd273e-83d2-4c79-a636-ab2b33e46117"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 10:56:51.380641 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.380575 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02cd273e-83d2-4c79-a636-ab2b33e46117-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "02cd273e-83d2-4c79-a636-ab2b33e46117" (UID: "02cd273e-83d2-4c79-a636-ab2b33e46117"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:56:51.380641 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.380599 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02cd273e-83d2-4c79-a636-ab2b33e46117-kube-api-access-dg8j8" (OuterVolumeSpecName: "kube-api-access-dg8j8") pod "02cd273e-83d2-4c79-a636-ab2b33e46117" (UID: "02cd273e-83d2-4c79-a636-ab2b33e46117"). InnerVolumeSpecName "kube-api-access-dg8j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:56:51.479687 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.479600 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg8j8\" (UniqueName: \"kubernetes.io/projected/02cd273e-83d2-4c79-a636-ab2b33e46117-kube-api-access-dg8j8\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:56:51.479687 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.479628 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02cd273e-83d2-4c79-a636-ab2b33e46117-dshm\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:56:51.479687 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.479638 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02cd273e-83d2-4c79-a636-ab2b33e46117-tmp-dir\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:56:51.479687 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.479646 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02cd273e-83d2-4c79-a636-ab2b33e46117-tls-certs\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:56:51.866545 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.866472 2559 generic.go:356] "Generic (PLEG): container finished" podID="02cd273e-83d2-4c79-a636-ab2b33e46117" containerID="e3cf662d74c75475c27046f2785b8a6d00c8089c2ccf54ad9c87bc921b4bdd1d" exitCode=0 Sep 09 10:56:51.866545 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.866543 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" Sep 09 10:56:51.866787 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.866549 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" event={"ID":"02cd273e-83d2-4c79-a636-ab2b33e46117","Type":"ContainerDied","Data":"e3cf662d74c75475c27046f2785b8a6d00c8089c2ccf54ad9c87bc921b4bdd1d"} Sep 09 10:56:51.866787 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.866588 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2" event={"ID":"02cd273e-83d2-4c79-a636-ab2b33e46117","Type":"ContainerDied","Data":"4299ca973daf0eeb1cbe17c58b791cdec6922b58e741c18def0d195acfaef2fc"} Sep 09 10:56:51.866787 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.866605 2559 scope.go:117] "RemoveContainer" containerID="e3cf662d74c75475c27046f2785b8a6d00c8089c2ccf54ad9c87bc921b4bdd1d" Sep 09 10:56:51.867953 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.867889 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a03d9e-582jg" Sep 09 10:56:51.867953 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.867936 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a03d9e-582jg" event={"ID":"8bc3071f-e672-42cf-a1b5-15447cef32ad","Type":"ContainerDied","Data":"99201b66c7b62b1dee3ed3ce1003a6d23262c87b66000431ecb277f02b2da022"} Sep 09 10:56:51.868053 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.867986 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99201b66c7b62b1dee3ed3ce1003a6d23262c87b66000431ecb277f02b2da022" Sep 09 10:56:51.876868 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.876849 2559 scope.go:117] "RemoveContainer" containerID="e3cf662d74c75475c27046f2785b8a6d00c8089c2ccf54ad9c87bc921b4bdd1d" Sep 09 10:56:51.877169 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:56:51.877147 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3cf662d74c75475c27046f2785b8a6d00c8089c2ccf54ad9c87bc921b4bdd1d\": container with ID starting with e3cf662d74c75475c27046f2785b8a6d00c8089c2ccf54ad9c87bc921b4bdd1d not found: ID does not exist" containerID="e3cf662d74c75475c27046f2785b8a6d00c8089c2ccf54ad9c87bc921b4bdd1d" Sep 09 10:56:51.877231 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.877177 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3cf662d74c75475c27046f2785b8a6d00c8089c2ccf54ad9c87bc921b4bdd1d"} err="failed to get container status \"e3cf662d74c75475c27046f2785b8a6d00c8089c2ccf54ad9c87bc921b4bdd1d\": rpc error: code = NotFound desc = could not find container \"e3cf662d74c75475c27046f2785b8a6d00c8089c2ccf54ad9c87bc921b4bdd1d\": container with ID starting with e3cf662d74c75475c27046f2785b8a6d00c8089c2ccf54ad9c87bc921b4bdd1d not found: ID does not exist" Sep 09 10:56:51.890146 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.890121 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2"] Sep 09 10:56:51.896271 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:51.896247 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1eb21d/rate-test-model-1eb21d-kserve-f768b65bc-wgdv2"] Sep 09 10:56:52.334383 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:52.334353 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02cd273e-83d2-4c79-a636-ab2b33e46117" path="/var/lib/kubelet/pods/02cd273e-83d2-4c79-a636-ab2b33e46117/volumes" Sep 09 10:56:55.833745 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:55.833714 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-33d685-7c7cd7c479-rbhzn" Sep 09 10:56:58.344371 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.344333 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a03d9e-68676b798c-kbl2x"] Sep 09 10:56:58.345237 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.345197 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a03d9e-68676b798c-kbl2x" podUID="adf2f9d1-1c28-4603-84ef-62a908476619" containerName="payload-processing" containerID="cri-o://598589bea897912cb86f986800c995b4b54d3a4caffa6b252e1b459aff7a7c71" gracePeriod=30 Sep 09 10:56:58.356555 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.356525 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a03d9e-66f9d5fbb-r2pw8"] Sep 09 10:56:58.356832 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.356801 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a03d9e-66f9d5fbb-r2pw8" podUID="cafa056b-6f23-4185-b494-cb022cdaa26d" containerName="payload-pre-processing" containerID="cri-o://bc5d374b55f1e26ec8b292490ec984326aa05f6e5e1388fcd368d74802be14ba" gracePeriod=30 Sep 09 10:56:58.532200 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.529297 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a03d9e-582jg"] Sep 09 10:56:58.535337 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.535312 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a03d9e-582jg"] Sep 09 10:56:58.639913 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.639881 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a03d9e-68676b798c-kbl2x" Sep 09 10:56:58.643284 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.643257 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a03d9e-66f9d5fbb-r2pw8" Sep 09 10:56:58.736475 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.736435 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf2f9d1-1c28-4603-84ef-62a908476619-plugins-config-volume\") pod \"adf2f9d1-1c28-4603-84ef-62a908476619\" (UID: \"adf2f9d1-1c28-4603-84ef-62a908476619\") " Sep 09 10:56:58.736671 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.736516 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cafa056b-6f23-4185-b494-cb022cdaa26d-plugins-config-volume\") pod \"cafa056b-6f23-4185-b494-cb022cdaa26d\" (UID: \"cafa056b-6f23-4185-b494-cb022cdaa26d\") " Sep 09 10:56:58.736671 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.736544 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6mpf\" (UniqueName: \"kubernetes.io/projected/adf2f9d1-1c28-4603-84ef-62a908476619-kube-api-access-h6mpf\") pod \"adf2f9d1-1c28-4603-84ef-62a908476619\" (UID: \"adf2f9d1-1c28-4603-84ef-62a908476619\") " Sep 09 10:56:58.736671 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.736588 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbg2l\" (UniqueName: \"kubernetes.io/projected/cafa056b-6f23-4185-b494-cb022cdaa26d-kube-api-access-zbg2l\") pod \"cafa056b-6f23-4185-b494-cb022cdaa26d\" (UID: \"cafa056b-6f23-4185-b494-cb022cdaa26d\") " Sep 09 10:56:58.736923 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.736891 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adf2f9d1-1c28-4603-84ef-62a908476619-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "adf2f9d1-1c28-4603-84ef-62a908476619" (UID: "adf2f9d1-1c28-4603-84ef-62a908476619"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:56:58.736980 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.736932 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cafa056b-6f23-4185-b494-cb022cdaa26d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cafa056b-6f23-4185-b494-cb022cdaa26d" (UID: "cafa056b-6f23-4185-b494-cb022cdaa26d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:56:58.738696 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.738674 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adf2f9d1-1c28-4603-84ef-62a908476619-kube-api-access-h6mpf" (OuterVolumeSpecName: "kube-api-access-h6mpf") pod "adf2f9d1-1c28-4603-84ef-62a908476619" (UID: "adf2f9d1-1c28-4603-84ef-62a908476619"). InnerVolumeSpecName "kube-api-access-h6mpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:56:58.738883 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.738860 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cafa056b-6f23-4185-b494-cb022cdaa26d-kube-api-access-zbg2l" (OuterVolumeSpecName: "kube-api-access-zbg2l") pod "cafa056b-6f23-4185-b494-cb022cdaa26d" (UID: "cafa056b-6f23-4185-b494-cb022cdaa26d"). InnerVolumeSpecName "kube-api-access-zbg2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:56:58.837817 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.837778 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf2f9d1-1c28-4603-84ef-62a908476619-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:56:58.837817 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.837810 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cafa056b-6f23-4185-b494-cb022cdaa26d-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:56:58.837817 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.837820 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6mpf\" (UniqueName: \"kubernetes.io/projected/adf2f9d1-1c28-4603-84ef-62a908476619-kube-api-access-h6mpf\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:56:58.837817 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.837828 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbg2l\" (UniqueName: \"kubernetes.io/projected/cafa056b-6f23-4185-b494-cb022cdaa26d-kube-api-access-zbg2l\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:56:58.905993 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.905910 2559 generic.go:356] "Generic (PLEG): container finished" podID="adf2f9d1-1c28-4603-84ef-62a908476619" containerID="598589bea897912cb86f986800c995b4b54d3a4caffa6b252e1b459aff7a7c71" exitCode=0 Sep 09 10:56:58.905993 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.905957 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a03d9e-68676b798c-kbl2x" Sep 09 10:56:58.906189 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.906005 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a03d9e-68676b798c-kbl2x" event={"ID":"adf2f9d1-1c28-4603-84ef-62a908476619","Type":"ContainerDied","Data":"598589bea897912cb86f986800c995b4b54d3a4caffa6b252e1b459aff7a7c71"} Sep 09 10:56:58.906189 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.906052 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a03d9e-68676b798c-kbl2x" event={"ID":"adf2f9d1-1c28-4603-84ef-62a908476619","Type":"ContainerDied","Data":"66a73de2068ab1d58651912a66b03b67be0fb9e8ae7852560f47e928c2536e78"} Sep 09 10:56:58.906189 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.906077 2559 scope.go:117] "RemoveContainer" containerID="598589bea897912cb86f986800c995b4b54d3a4caffa6b252e1b459aff7a7c71" Sep 09 10:56:58.907260 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.907241 2559 generic.go:356] "Generic (PLEG): container finished" podID="cafa056b-6f23-4185-b494-cb022cdaa26d" containerID="bc5d374b55f1e26ec8b292490ec984326aa05f6e5e1388fcd368d74802be14ba" exitCode=0 Sep 09 10:56:58.907347 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.907279 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a03d9e-66f9d5fbb-r2pw8" event={"ID":"cafa056b-6f23-4185-b494-cb022cdaa26d","Type":"ContainerDied","Data":"bc5d374b55f1e26ec8b292490ec984326aa05f6e5e1388fcd368d74802be14ba"} Sep 09 10:56:58.907347 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.907312 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a03d9e-66f9d5fbb-r2pw8" Sep 09 10:56:58.907425 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.907319 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a03d9e-66f9d5fbb-r2pw8" event={"ID":"cafa056b-6f23-4185-b494-cb022cdaa26d","Type":"ContainerDied","Data":"80a6578459b08af05af3fa53b9b83a5c68a85bd1bbbd9d90d2f20ff95f34a77a"} Sep 09 10:56:58.915710 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.915684 2559 scope.go:117] "RemoveContainer" containerID="598589bea897912cb86f986800c995b4b54d3a4caffa6b252e1b459aff7a7c71" Sep 09 10:56:58.915997 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:56:58.915977 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"598589bea897912cb86f986800c995b4b54d3a4caffa6b252e1b459aff7a7c71\": container with ID starting with 598589bea897912cb86f986800c995b4b54d3a4caffa6b252e1b459aff7a7c71 not found: ID does not exist" containerID="598589bea897912cb86f986800c995b4b54d3a4caffa6b252e1b459aff7a7c71" Sep 09 10:56:58.916067 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.916005 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"598589bea897912cb86f986800c995b4b54d3a4caffa6b252e1b459aff7a7c71"} err="failed to get container status \"598589bea897912cb86f986800c995b4b54d3a4caffa6b252e1b459aff7a7c71\": rpc error: code = NotFound desc = could not find container \"598589bea897912cb86f986800c995b4b54d3a4caffa6b252e1b459aff7a7c71\": container with ID starting with 598589bea897912cb86f986800c995b4b54d3a4caffa6b252e1b459aff7a7c71 not found: ID does not exist" Sep 09 10:56:58.916067 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.916021 2559 scope.go:117] "RemoveContainer" containerID="bc5d374b55f1e26ec8b292490ec984326aa05f6e5e1388fcd368d74802be14ba" Sep 09 10:56:58.924370 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.924350 2559 scope.go:117] "RemoveContainer" containerID="bc5d374b55f1e26ec8b292490ec984326aa05f6e5e1388fcd368d74802be14ba" Sep 09 10:56:58.924643 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:56:58.924625 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc5d374b55f1e26ec8b292490ec984326aa05f6e5e1388fcd368d74802be14ba\": container with ID starting with bc5d374b55f1e26ec8b292490ec984326aa05f6e5e1388fcd368d74802be14ba not found: ID does not exist" containerID="bc5d374b55f1e26ec8b292490ec984326aa05f6e5e1388fcd368d74802be14ba" Sep 09 10:56:58.924686 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.924649 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc5d374b55f1e26ec8b292490ec984326aa05f6e5e1388fcd368d74802be14ba"} err="failed to get container status \"bc5d374b55f1e26ec8b292490ec984326aa05f6e5e1388fcd368d74802be14ba\": rpc error: code = NotFound desc = could not find container \"bc5d374b55f1e26ec8b292490ec984326aa05f6e5e1388fcd368d74802be14ba\": container with ID starting with bc5d374b55f1e26ec8b292490ec984326aa05f6e5e1388fcd368d74802be14ba not found: ID does not exist" Sep 09 10:56:58.931964 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.931941 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a03d9e-68676b798c-kbl2x"] Sep 09 10:56:58.936858 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.936837 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a03d9e-68676b798c-kbl2x"] Sep 09 10:56:58.951399 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.951365 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a03d9e-66f9d5fbb-r2pw8"] Sep 09 10:56:58.954959 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:56:58.954931 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a03d9e-66f9d5fbb-r2pw8"] Sep 09 10:57:00.335858 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:00.335823 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bc3071f-e672-42cf-a1b5-15447cef32ad" path="/var/lib/kubelet/pods/8bc3071f-e672-42cf-a1b5-15447cef32ad/volumes" Sep 09 10:57:00.336211 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:00.336168 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adf2f9d1-1c28-4603-84ef-62a908476619" path="/var/lib/kubelet/pods/adf2f9d1-1c28-4603-84ef-62a908476619/volumes" Sep 09 10:57:00.336478 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:00.336465 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cafa056b-6f23-4185-b494-cb022cdaa26d" path="/var/lib/kubelet/pods/cafa056b-6f23-4185-b494-cb022cdaa26d/volumes" Sep 09 10:57:12.362139 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.362106 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz"] Sep 09 10:57:12.362638 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.362475 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bc3071f-e672-42cf-a1b5-15447cef32ad" containerName="revoke-keys" Sep 09 10:57:12.362638 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.362495 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bc3071f-e672-42cf-a1b5-15447cef32ad" containerName="revoke-keys" Sep 09 10:57:12.362638 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.362527 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adf2f9d1-1c28-4603-84ef-62a908476619" containerName="payload-processing" Sep 09 10:57:12.362638 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.362534 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="adf2f9d1-1c28-4603-84ef-62a908476619" containerName="payload-processing" Sep 09 10:57:12.362638 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.362543 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02cd273e-83d2-4c79-a636-ab2b33e46117" containerName="main" Sep 09 10:57:12.362638 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.362549 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="02cd273e-83d2-4c79-a636-ab2b33e46117" containerName="main" Sep 09 10:57:12.362638 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.362557 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cafa056b-6f23-4185-b494-cb022cdaa26d" containerName="payload-pre-processing" Sep 09 10:57:12.362638 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.362564 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cafa056b-6f23-4185-b494-cb022cdaa26d" containerName="payload-pre-processing" Sep 09 10:57:12.362638 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.362618 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="02cd273e-83d2-4c79-a636-ab2b33e46117" containerName="main" Sep 09 10:57:12.362638 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.362628 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bc3071f-e672-42cf-a1b5-15447cef32ad" containerName="revoke-keys" Sep 09 10:57:12.362638 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.362636 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="adf2f9d1-1c28-4603-84ef-62a908476619" containerName="payload-processing" Sep 09 10:57:12.362638 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.362641 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cafa056b-6f23-4185-b494-cb022cdaa26d" containerName="payload-pre-processing" Sep 09 10:57:12.365374 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.365357 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz" Sep 09 10:57:12.368048 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.368024 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c1bfbb-dockercfg-glmwv\"" Sep 09 10:57:12.368672 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.368650 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c1bfbb\"" Sep 09 10:57:12.378743 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.378713 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz"] Sep 09 10:57:12.412052 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.412021 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c1bfbb-f47f7b7bb-t69bz"] Sep 09 10:57:12.415309 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.415291 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1bfbb-f47f7b7bb-t69bz" Sep 09 10:57:12.428187 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.428159 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c1bfbb-f47f7b7bb-t69bz"] Sep 09 10:57:12.464400 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.464363 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd21fb4-40d9-4d30-9b04-9ed81a2b540e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz\" (UID: \"7cd21fb4-40d9-4d30-9b04-9ed81a2b540e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz" Sep 09 10:57:12.464609 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.464439 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/775ee7e4-76f8-4cb1-af48-f8b82716110a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c1bfbb-f47f7b7bb-t69bz\" (UID: \"775ee7e4-76f8-4cb1-af48-f8b82716110a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1bfbb-f47f7b7bb-t69bz" Sep 09 10:57:12.464609 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.464483 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltwmh\" (UniqueName: \"kubernetes.io/projected/775ee7e4-76f8-4cb1-af48-f8b82716110a-kube-api-access-ltwmh\") pod \"payload-pre-processing-e2e-ipp-b-c1bfbb-f47f7b7bb-t69bz\" (UID: \"775ee7e4-76f8-4cb1-af48-f8b82716110a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1bfbb-f47f7b7bb-t69bz" Sep 09 10:57:12.464609 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.464582 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6qlb\" (UniqueName: \"kubernetes.io/projected/7cd21fb4-40d9-4d30-9b04-9ed81a2b540e-kube-api-access-k6qlb\") pod \"payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz\" (UID: \"7cd21fb4-40d9-4d30-9b04-9ed81a2b540e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz" Sep 09 10:57:12.565748 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.565706 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltwmh\" (UniqueName: \"kubernetes.io/projected/775ee7e4-76f8-4cb1-af48-f8b82716110a-kube-api-access-ltwmh\") pod \"payload-pre-processing-e2e-ipp-b-c1bfbb-f47f7b7bb-t69bz\" (UID: \"775ee7e4-76f8-4cb1-af48-f8b82716110a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1bfbb-f47f7b7bb-t69bz" Sep 09 10:57:12.565942 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.565776 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6qlb\" (UniqueName: \"kubernetes.io/projected/7cd21fb4-40d9-4d30-9b04-9ed81a2b540e-kube-api-access-k6qlb\") pod \"payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz\" (UID: \"7cd21fb4-40d9-4d30-9b04-9ed81a2b540e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz" Sep 09 10:57:12.565942 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.565827 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd21fb4-40d9-4d30-9b04-9ed81a2b540e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz\" (UID: \"7cd21fb4-40d9-4d30-9b04-9ed81a2b540e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz" Sep 09 10:57:12.565942 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.565889 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/775ee7e4-76f8-4cb1-af48-f8b82716110a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c1bfbb-f47f7b7bb-t69bz\" (UID: \"775ee7e4-76f8-4cb1-af48-f8b82716110a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1bfbb-f47f7b7bb-t69bz" Sep 09 10:57:12.566674 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.566646 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd21fb4-40d9-4d30-9b04-9ed81a2b540e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz\" (UID: \"7cd21fb4-40d9-4d30-9b04-9ed81a2b540e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz" Sep 09 10:57:12.566781 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.566653 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/775ee7e4-76f8-4cb1-af48-f8b82716110a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c1bfbb-f47f7b7bb-t69bz\" (UID: \"775ee7e4-76f8-4cb1-af48-f8b82716110a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1bfbb-f47f7b7bb-t69bz" Sep 09 10:57:12.576029 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.576001 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltwmh\" (UniqueName: \"kubernetes.io/projected/775ee7e4-76f8-4cb1-af48-f8b82716110a-kube-api-access-ltwmh\") pod \"payload-pre-processing-e2e-ipp-b-c1bfbb-f47f7b7bb-t69bz\" (UID: \"775ee7e4-76f8-4cb1-af48-f8b82716110a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1bfbb-f47f7b7bb-t69bz" Sep 09 10:57:12.576160 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.576001 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6qlb\" (UniqueName: \"kubernetes.io/projected/7cd21fb4-40d9-4d30-9b04-9ed81a2b540e-kube-api-access-k6qlb\") pod \"payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz\" (UID: \"7cd21fb4-40d9-4d30-9b04-9ed81a2b540e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz" Sep 09 10:57:12.675852 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.675753 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz" Sep 09 10:57:12.726076 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.725909 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1bfbb-f47f7b7bb-t69bz" Sep 09 10:57:12.951029 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.950902 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz"] Sep 09 10:57:12.953326 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:57:12.953293 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd21fb4_40d9_4d30_9b04_9ed81a2b540e.slice/crio-a9f04ba9f8a679f55669d032dae2f34045f5d481357d14192ce4a4ccb0747cc0 WatchSource:0}: Error finding container a9f04ba9f8a679f55669d032dae2f34045f5d481357d14192ce4a4ccb0747cc0: Status 404 returned error can't find the container with id a9f04ba9f8a679f55669d032dae2f34045f5d481357d14192ce4a4ccb0747cc0 Sep 09 10:57:12.982508 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:12.982455 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz" event={"ID":"7cd21fb4-40d9-4d30-9b04-9ed81a2b540e","Type":"ContainerStarted","Data":"a9f04ba9f8a679f55669d032dae2f34045f5d481357d14192ce4a4ccb0747cc0"} Sep 09 10:57:13.011202 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:13.011173 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c1bfbb-f47f7b7bb-t69bz"] Sep 09 10:57:13.013031 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:57:13.013007 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775ee7e4_76f8_4cb1_af48_f8b82716110a.slice/crio-0864d92f41b27d315820f989c2ebbc9cfbf0a6a5d5aef41967be957a6216e830 WatchSource:0}: Error finding container 0864d92f41b27d315820f989c2ebbc9cfbf0a6a5d5aef41967be957a6216e830: Status 404 returned error can't find the container with id 0864d92f41b27d315820f989c2ebbc9cfbf0a6a5d5aef41967be957a6216e830 Sep 09 10:57:13.990098 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:13.990054 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1bfbb-f47f7b7bb-t69bz" event={"ID":"775ee7e4-76f8-4cb1-af48-f8b82716110a","Type":"ContainerStarted","Data":"fee9d50c70b4fe6026b1508c9b6ea6d001e18989e207a10b5a240cc6e16186ca"} Sep 09 10:57:13.990592 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:13.990168 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1bfbb-f47f7b7bb-t69bz" Sep 09 10:57:13.990592 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:13.990200 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1bfbb-f47f7b7bb-t69bz" event={"ID":"775ee7e4-76f8-4cb1-af48-f8b82716110a","Type":"ContainerStarted","Data":"0864d92f41b27d315820f989c2ebbc9cfbf0a6a5d5aef41967be957a6216e830"} Sep 09 10:57:13.991867 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:13.991835 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz" event={"ID":"7cd21fb4-40d9-4d30-9b04-9ed81a2b540e","Type":"ContainerStarted","Data":"11b51c4d3c9a39682e00ab3516b48709336b3e87affa735d93a89a1df7cbe168"} Sep 09 10:57:13.991990 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:13.991970 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz" Sep 09 10:57:14.008048 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:14.008000 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1bfbb-f47f7b7bb-t69bz" podStartSLOduration=2.007987523 podStartE2EDuration="2.007987523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:57:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:57:14.00698207 +0000 UTC m=+1874.181584396" watchObservedRunningTime="2026-09-09 10:57:14.007987523 +0000 UTC m=+1874.182589848" Sep 09 10:57:14.025986 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:14.025936 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz" podStartSLOduration=2.025920511 podStartE2EDuration="2.025920511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:57:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:57:14.023830753 +0000 UTC m=+1874.198433078" watchObservedRunningTime="2026-09-09 10:57:14.025920511 +0000 UTC m=+1874.200522833" Sep 09 10:57:25.000131 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:25.000047 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz" Sep 09 10:57:25.000665 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:25.000137 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1bfbb-f47f7b7bb-t69bz" Sep 09 10:57:40.856251 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:40.856218 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-x4njj_1cdfe780-23e6-4fa2-bb10-dd57f4c5c25b/payload-processing/0.log" Sep 09 10:57:40.981218 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:57:40.981187 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz_7cd21fb4-40d9-4d30-9b04-9ed81a2b540e/payload-processing/0.log" Sep 09 10:58:21.769955 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:21.769924 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-33d685-7c7cd7c479-rbhzn_60aea43a-6bf8-406d-80e2-61b46e25c98f/payload-processing/0.log" Sep 09 10:58:21.901020 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:21.900991 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-x4njj_1cdfe780-23e6-4fa2-bb10-dd57f4c5c25b/payload-processing/0.log" Sep 09 10:58:22.023448 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:22.023362 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz_7cd21fb4-40d9-4d30-9b04-9ed81a2b540e/payload-processing/0.log" Sep 09 10:58:27.784439 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:27.784399 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f7ff8b-c9d44cdf8-t6t46"] Sep 09 10:58:27.788044 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:27.788023 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f7ff8b-c9d44cdf8-t6t46" Sep 09 10:58:27.791110 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:27.791087 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-f7ff8b\"" Sep 09 10:58:27.791246 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:27.791130 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-f7ff8b-dockercfg-pzdks\"" Sep 09 10:58:27.800981 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:27.800953 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f7ff8b-c9d44cdf8-t6t46"] Sep 09 10:58:27.803420 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:27.803388 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f7ff8b-866bb5656f-gsbkw"] Sep 09 10:58:27.807214 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:27.807194 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f7ff8b-866bb5656f-gsbkw" Sep 09 10:58:27.817924 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:27.817894 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f7ff8b-866bb5656f-gsbkw"] Sep 09 10:58:27.927309 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:27.927272 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87c55d2-4dbf-49d0-9174-f8ad8220610f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-f7ff8b-866bb5656f-gsbkw\" (UID: \"d87c55d2-4dbf-49d0-9174-f8ad8220610f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f7ff8b-866bb5656f-gsbkw" Sep 09 10:58:27.927724 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:27.927426 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e0f57e3-f642-40b4-bdb8-acc008f311d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-f7ff8b-c9d44cdf8-t6t46\" (UID: \"8e0f57e3-f642-40b4-bdb8-acc008f311d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f7ff8b-c9d44cdf8-t6t46" Sep 09 10:58:27.927724 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:27.927466 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxlm7\" (UniqueName: \"kubernetes.io/projected/8e0f57e3-f642-40b4-bdb8-acc008f311d8-kube-api-access-kxlm7\") pod \"payload-pre-processing-e2e-ipp-cleanup-f7ff8b-c9d44cdf8-t6t46\" (UID: \"8e0f57e3-f642-40b4-bdb8-acc008f311d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f7ff8b-c9d44cdf8-t6t46" Sep 09 10:58:27.927724 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:27.927521 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6th6p\" (UniqueName: \"kubernetes.io/projected/d87c55d2-4dbf-49d0-9174-f8ad8220610f-kube-api-access-6th6p\") pod \"payload-processing-e2e-ipp-cleanup-f7ff8b-866bb5656f-gsbkw\" (UID: \"d87c55d2-4dbf-49d0-9174-f8ad8220610f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f7ff8b-866bb5656f-gsbkw" Sep 09 10:58:28.028757 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:28.028702 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87c55d2-4dbf-49d0-9174-f8ad8220610f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-f7ff8b-866bb5656f-gsbkw\" (UID: \"d87c55d2-4dbf-49d0-9174-f8ad8220610f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f7ff8b-866bb5656f-gsbkw" Sep 09 10:58:28.028980 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:28.028830 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e0f57e3-f642-40b4-bdb8-acc008f311d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-f7ff8b-c9d44cdf8-t6t46\" (UID: \"8e0f57e3-f642-40b4-bdb8-acc008f311d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f7ff8b-c9d44cdf8-t6t46" Sep 09 10:58:28.028980 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:28.028858 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxlm7\" (UniqueName: \"kubernetes.io/projected/8e0f57e3-f642-40b4-bdb8-acc008f311d8-kube-api-access-kxlm7\") pod \"payload-pre-processing-e2e-ipp-cleanup-f7ff8b-c9d44cdf8-t6t46\" (UID: \"8e0f57e3-f642-40b4-bdb8-acc008f311d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f7ff8b-c9d44cdf8-t6t46" Sep 09 10:58:28.028980 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:28.028877 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6th6p\" (UniqueName: \"kubernetes.io/projected/d87c55d2-4dbf-49d0-9174-f8ad8220610f-kube-api-access-6th6p\") pod \"payload-processing-e2e-ipp-cleanup-f7ff8b-866bb5656f-gsbkw\" (UID: \"d87c55d2-4dbf-49d0-9174-f8ad8220610f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f7ff8b-866bb5656f-gsbkw" Sep 09 10:58:28.029599 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:28.029575 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87c55d2-4dbf-49d0-9174-f8ad8220610f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-f7ff8b-866bb5656f-gsbkw\" (UID: \"d87c55d2-4dbf-49d0-9174-f8ad8220610f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f7ff8b-866bb5656f-gsbkw" Sep 09 10:58:28.029692 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:28.029638 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e0f57e3-f642-40b4-bdb8-acc008f311d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-f7ff8b-c9d44cdf8-t6t46\" (UID: \"8e0f57e3-f642-40b4-bdb8-acc008f311d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f7ff8b-c9d44cdf8-t6t46" Sep 09 10:58:28.038211 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:28.038138 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxlm7\" (UniqueName: \"kubernetes.io/projected/8e0f57e3-f642-40b4-bdb8-acc008f311d8-kube-api-access-kxlm7\") pod \"payload-pre-processing-e2e-ipp-cleanup-f7ff8b-c9d44cdf8-t6t46\" (UID: \"8e0f57e3-f642-40b4-bdb8-acc008f311d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f7ff8b-c9d44cdf8-t6t46" Sep 09 10:58:28.038211 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:28.038160 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6th6p\" (UniqueName: \"kubernetes.io/projected/d87c55d2-4dbf-49d0-9174-f8ad8220610f-kube-api-access-6th6p\") pod \"payload-processing-e2e-ipp-cleanup-f7ff8b-866bb5656f-gsbkw\" (UID: \"d87c55d2-4dbf-49d0-9174-f8ad8220610f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f7ff8b-866bb5656f-gsbkw" Sep 09 10:58:28.099213 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:28.099165 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f7ff8b-c9d44cdf8-t6t46" Sep 09 10:58:28.119564 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:28.119532 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f7ff8b-866bb5656f-gsbkw" Sep 09 10:58:28.383429 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:28.383314 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f7ff8b-c9d44cdf8-t6t46"] Sep 09 10:58:28.385869 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:58:28.385842 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e0f57e3_f642_40b4_bdb8_acc008f311d8.slice/crio-78fc962beac032c21466adb26bf283d155dd08bca4db7e45ab359749c2cf52eb WatchSource:0}: Error finding container 78fc962beac032c21466adb26bf283d155dd08bca4db7e45ab359749c2cf52eb: Status 404 returned error can't find the container with id 78fc962beac032c21466adb26bf283d155dd08bca4db7e45ab359749c2cf52eb Sep 09 10:58:28.415139 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:28.415104 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f7ff8b-866bb5656f-gsbkw"] Sep 09 10:58:28.417539 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:58:28.417514 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd87c55d2_4dbf_49d0_9174_f8ad8220610f.slice/crio-8a84702d3565eb789c6e5823f8aac128f940acf59bf2bc084afa3851b51c0594 WatchSource:0}: Error finding container 8a84702d3565eb789c6e5823f8aac128f940acf59bf2bc084afa3851b51c0594: Status 404 returned error can't find the container with id 8a84702d3565eb789c6e5823f8aac128f940acf59bf2bc084afa3851b51c0594 Sep 09 10:58:29.386955 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:29.386919 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f7ff8b-866bb5656f-gsbkw" event={"ID":"d87c55d2-4dbf-49d0-9174-f8ad8220610f","Type":"ContainerStarted","Data":"bcf3559d61a9b6c37577b6a3076aabfc5238234bf8120f096f52219f6261484e"} Sep 09 10:58:29.387461 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:29.386983 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f7ff8b-866bb5656f-gsbkw" event={"ID":"d87c55d2-4dbf-49d0-9174-f8ad8220610f","Type":"ContainerStarted","Data":"8a84702d3565eb789c6e5823f8aac128f940acf59bf2bc084afa3851b51c0594"} Sep 09 10:58:29.387461 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:29.387084 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f7ff8b-866bb5656f-gsbkw" Sep 09 10:58:29.388846 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:29.388819 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f7ff8b-c9d44cdf8-t6t46" event={"ID":"8e0f57e3-f642-40b4-bdb8-acc008f311d8","Type":"ContainerStarted","Data":"80d10926a1959e0f78391d1fa7bf80440ab4f310338b6cdbeac55011fbdaa3bf"} Sep 09 10:58:29.388973 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:29.388870 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f7ff8b-c9d44cdf8-t6t46" event={"ID":"8e0f57e3-f642-40b4-bdb8-acc008f311d8","Type":"ContainerStarted","Data":"78fc962beac032c21466adb26bf283d155dd08bca4db7e45ab359749c2cf52eb"} Sep 09 10:58:29.388973 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:29.388899 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f7ff8b-c9d44cdf8-t6t46" Sep 09 10:58:29.405543 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:29.405476 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f7ff8b-866bb5656f-gsbkw" podStartSLOduration=2.405459453 podStartE2EDuration="2.405459453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:58:29.40370301 +0000 UTC m=+1949.578305339" watchObservedRunningTime="2026-09-09 10:58:29.405459453 +0000 UTC m=+1949.580061775" Sep 09 10:58:29.419769 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:29.419715 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f7ff8b-c9d44cdf8-t6t46" podStartSLOduration=2.419699278 podStartE2EDuration="2.419699278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:58:29.419129926 +0000 UTC m=+1949.593732252" watchObservedRunningTime="2026-09-09 10:58:29.419699278 +0000 UTC m=+1949.594301603" Sep 09 10:58:40.402207 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:40.402171 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f7ff8b-c9d44cdf8-t6t46" Sep 09 10:58:40.402739 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:40.402354 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f7ff8b-866bb5656f-gsbkw" Sep 09 10:58:51.439448 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:51.439410 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f7ff8b-z4tmv"] Sep 09 10:58:51.443056 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:51.443039 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f7ff8b-z4tmv" Sep 09 10:58:51.446640 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:51.446618 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kq7qv\"" Sep 09 10:58:51.446940 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:51.446921 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 10:58:51.461117 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:51.461085 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f7ff8b-z4tmv"] Sep 09 10:58:51.542038 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:51.541998 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f59f4115-1ff3-411f-bfb0-267eb87a7a73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f7ff8b-z4tmv\" (UID: \"f59f4115-1ff3-411f-bfb0-267eb87a7a73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f7ff8b-z4tmv" Sep 09 10:58:51.643397 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:51.643357 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f59f4115-1ff3-411f-bfb0-267eb87a7a73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f7ff8b-z4tmv\" (UID: \"f59f4115-1ff3-411f-bfb0-267eb87a7a73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f7ff8b-z4tmv" Sep 09 10:58:51.644026 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:51.644003 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f59f4115-1ff3-411f-bfb0-267eb87a7a73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f7ff8b-z4tmv\" (UID: \"f59f4115-1ff3-411f-bfb0-267eb87a7a73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f7ff8b-z4tmv" Sep 09 10:58:51.753766 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:51.753664 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f7ff8b-z4tmv" Sep 09 10:58:52.027659 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:52.027635 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f7ff8b-z4tmv"] Sep 09 10:58:52.029836 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:58:52.029806 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59f4115_1ff3_411f_bfb0_267eb87a7a73.slice/crio-0f6f6e0f8202cdbf1e17d839709d8dd4464a49533992a5cde455337707061197 WatchSource:0}: Error finding container 0f6f6e0f8202cdbf1e17d839709d8dd4464a49533992a5cde455337707061197: Status 404 returned error can't find the container with id 0f6f6e0f8202cdbf1e17d839709d8dd4464a49533992a5cde455337707061197 Sep 09 10:58:52.517388 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:52.517352 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f7ff8b-z4tmv" event={"ID":"f59f4115-1ff3-411f-bfb0-267eb87a7a73","Type":"ContainerStarted","Data":"1ebbd3aac841af9866d03c0aa094422a311938398eecedd5c68fb26c9b6b3f78"} Sep 09 10:58:52.517812 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:52.517405 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f7ff8b-z4tmv" event={"ID":"f59f4115-1ff3-411f-bfb0-267eb87a7a73","Type":"ContainerStarted","Data":"0f6f6e0f8202cdbf1e17d839709d8dd4464a49533992a5cde455337707061197"} Sep 09 10:58:52.538641 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:52.538586 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f7ff8b-z4tmv" podStartSLOduration=1.538569677 podStartE2EDuration="1.538569677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:58:52.536471185 +0000 UTC m=+1972.711073511" watchObservedRunningTime="2026-09-09 10:58:52.538569677 +0000 UTC m=+1972.713172058" Sep 09 10:58:53.524025 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:53.523986 2559 generic.go:356] "Generic (PLEG): container finished" podID="f59f4115-1ff3-411f-bfb0-267eb87a7a73" containerID="1ebbd3aac841af9866d03c0aa094422a311938398eecedd5c68fb26c9b6b3f78" exitCode=0 Sep 09 10:58:53.524482 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:53.524070 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f7ff8b-z4tmv" event={"ID":"f59f4115-1ff3-411f-bfb0-267eb87a7a73","Type":"ContainerDied","Data":"1ebbd3aac841af9866d03c0aa094422a311938398eecedd5c68fb26c9b6b3f78"} Sep 09 10:58:54.666310 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:54.666280 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f7ff8b-z4tmv" Sep 09 10:58:54.774147 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:54.774108 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f59f4115-1ff3-411f-bfb0-267eb87a7a73-maas-api-service-ca\") pod \"f59f4115-1ff3-411f-bfb0-267eb87a7a73\" (UID: \"f59f4115-1ff3-411f-bfb0-267eb87a7a73\") " Sep 09 10:58:54.774477 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:54.774453 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f59f4115-1ff3-411f-bfb0-267eb87a7a73-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f59f4115-1ff3-411f-bfb0-267eb87a7a73" (UID: "f59f4115-1ff3-411f-bfb0-267eb87a7a73"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:58:54.874841 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:54.874802 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f59f4115-1ff3-411f-bfb0-267eb87a7a73-maas-api-service-ca\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:58:55.535543 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:55.535470 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f7ff8b-z4tmv" event={"ID":"f59f4115-1ff3-411f-bfb0-267eb87a7a73","Type":"ContainerDied","Data":"0f6f6e0f8202cdbf1e17d839709d8dd4464a49533992a5cde455337707061197"} Sep 09 10:58:55.535723 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:55.535588 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f6f6e0f8202cdbf1e17d839709d8dd4464a49533992a5cde455337707061197" Sep 09 10:58:55.535723 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:55.535521 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f7ff8b-z4tmv" Sep 09 10:58:57.766947 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:57.766911 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f7ff8b-866bb5656f-gsbkw"] Sep 09 10:58:57.767435 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:57.767147 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f7ff8b-866bb5656f-gsbkw" podUID="d87c55d2-4dbf-49d0-9174-f8ad8220610f" containerName="payload-processing" containerID="cri-o://bcf3559d61a9b6c37577b6a3076aabfc5238234bf8120f096f52219f6261484e" gracePeriod=30 Sep 09 10:58:57.778659 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:57.778623 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f7ff8b-c9d44cdf8-t6t46"] Sep 09 10:58:57.779327 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:57.779099 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f7ff8b-c9d44cdf8-t6t46" podUID="8e0f57e3-f642-40b4-bdb8-acc008f311d8" containerName="payload-pre-processing" containerID="cri-o://80d10926a1959e0f78391d1fa7bf80440ab4f310338b6cdbeac55011fbdaa3bf" gracePeriod=30 Sep 09 10:58:57.959613 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:57.959571 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f7ff8b-z4tmv"] Sep 09 10:58:57.961523 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:57.961481 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f7ff8b-z4tmv"] Sep 09 10:58:58.040649 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.040623 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f7ff8b-866bb5656f-gsbkw" Sep 09 10:58:58.076133 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.076105 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f7ff8b-c9d44cdf8-t6t46" Sep 09 10:58:58.102474 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.102420 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87c55d2-4dbf-49d0-9174-f8ad8220610f-plugins-config-volume\") pod \"d87c55d2-4dbf-49d0-9174-f8ad8220610f\" (UID: \"d87c55d2-4dbf-49d0-9174-f8ad8220610f\") " Sep 09 10:58:58.102653 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.102483 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6th6p\" (UniqueName: \"kubernetes.io/projected/d87c55d2-4dbf-49d0-9174-f8ad8220610f-kube-api-access-6th6p\") pod \"d87c55d2-4dbf-49d0-9174-f8ad8220610f\" (UID: \"d87c55d2-4dbf-49d0-9174-f8ad8220610f\") " Sep 09 10:58:58.102821 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.102799 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d87c55d2-4dbf-49d0-9174-f8ad8220610f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d87c55d2-4dbf-49d0-9174-f8ad8220610f" (UID: "d87c55d2-4dbf-49d0-9174-f8ad8220610f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:58:58.102885 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.102871 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87c55d2-4dbf-49d0-9174-f8ad8220610f-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:58:58.104942 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.104915 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d87c55d2-4dbf-49d0-9174-f8ad8220610f-kube-api-access-6th6p" (OuterVolumeSpecName: "kube-api-access-6th6p") pod "d87c55d2-4dbf-49d0-9174-f8ad8220610f" (UID: "d87c55d2-4dbf-49d0-9174-f8ad8220610f"). InnerVolumeSpecName "kube-api-access-6th6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:58:58.203641 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.203607 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxlm7\" (UniqueName: \"kubernetes.io/projected/8e0f57e3-f642-40b4-bdb8-acc008f311d8-kube-api-access-kxlm7\") pod \"8e0f57e3-f642-40b4-bdb8-acc008f311d8\" (UID: \"8e0f57e3-f642-40b4-bdb8-acc008f311d8\") " Sep 09 10:58:58.203818 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.203670 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e0f57e3-f642-40b4-bdb8-acc008f311d8-plugins-config-volume\") pod \"8e0f57e3-f642-40b4-bdb8-acc008f311d8\" (UID: \"8e0f57e3-f642-40b4-bdb8-acc008f311d8\") " Sep 09 10:58:58.203818 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.203806 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6th6p\" (UniqueName: \"kubernetes.io/projected/d87c55d2-4dbf-49d0-9174-f8ad8220610f-kube-api-access-6th6p\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:58:58.204003 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.203976 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e0f57e3-f642-40b4-bdb8-acc008f311d8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8e0f57e3-f642-40b4-bdb8-acc008f311d8" (UID: "8e0f57e3-f642-40b4-bdb8-acc008f311d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:58:58.205763 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.205737 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e0f57e3-f642-40b4-bdb8-acc008f311d8-kube-api-access-kxlm7" (OuterVolumeSpecName: "kube-api-access-kxlm7") pod "8e0f57e3-f642-40b4-bdb8-acc008f311d8" (UID: "8e0f57e3-f642-40b4-bdb8-acc008f311d8"). InnerVolumeSpecName "kube-api-access-kxlm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:58:58.305325 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.305226 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxlm7\" (UniqueName: \"kubernetes.io/projected/8e0f57e3-f642-40b4-bdb8-acc008f311d8-kube-api-access-kxlm7\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:58:58.305325 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.305263 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e0f57e3-f642-40b4-bdb8-acc008f311d8-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:58:58.334239 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.334207 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f59f4115-1ff3-411f-bfb0-267eb87a7a73" path="/var/lib/kubelet/pods/f59f4115-1ff3-411f-bfb0-267eb87a7a73/volumes" Sep 09 10:58:58.552914 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.552880 2559 generic.go:356] "Generic (PLEG): container finished" podID="d87c55d2-4dbf-49d0-9174-f8ad8220610f" containerID="bcf3559d61a9b6c37577b6a3076aabfc5238234bf8120f096f52219f6261484e" exitCode=0 Sep 09 10:58:58.552914 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.552923 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f7ff8b-866bb5656f-gsbkw" Sep 09 10:58:58.553162 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.552969 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f7ff8b-866bb5656f-gsbkw" event={"ID":"d87c55d2-4dbf-49d0-9174-f8ad8220610f","Type":"ContainerDied","Data":"bcf3559d61a9b6c37577b6a3076aabfc5238234bf8120f096f52219f6261484e"} Sep 09 10:58:58.553162 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.553020 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f7ff8b-866bb5656f-gsbkw" event={"ID":"d87c55d2-4dbf-49d0-9174-f8ad8220610f","Type":"ContainerDied","Data":"8a84702d3565eb789c6e5823f8aac128f940acf59bf2bc084afa3851b51c0594"} Sep 09 10:58:58.553162 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.553048 2559 scope.go:117] "RemoveContainer" containerID="bcf3559d61a9b6c37577b6a3076aabfc5238234bf8120f096f52219f6261484e" Sep 09 10:58:58.554385 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.554358 2559 generic.go:356] "Generic (PLEG): container finished" podID="8e0f57e3-f642-40b4-bdb8-acc008f311d8" containerID="80d10926a1959e0f78391d1fa7bf80440ab4f310338b6cdbeac55011fbdaa3bf" exitCode=0 Sep 09 10:58:58.554530 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.554399 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f7ff8b-c9d44cdf8-t6t46" Sep 09 10:58:58.554530 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.554418 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f7ff8b-c9d44cdf8-t6t46" event={"ID":"8e0f57e3-f642-40b4-bdb8-acc008f311d8","Type":"ContainerDied","Data":"80d10926a1959e0f78391d1fa7bf80440ab4f310338b6cdbeac55011fbdaa3bf"} Sep 09 10:58:58.554530 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.554450 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f7ff8b-c9d44cdf8-t6t46" event={"ID":"8e0f57e3-f642-40b4-bdb8-acc008f311d8","Type":"ContainerDied","Data":"78fc962beac032c21466adb26bf283d155dd08bca4db7e45ab359749c2cf52eb"} Sep 09 10:58:58.564855 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.564761 2559 scope.go:117] "RemoveContainer" containerID="bcf3559d61a9b6c37577b6a3076aabfc5238234bf8120f096f52219f6261484e" Sep 09 10:58:58.565267 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:58:58.565243 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcf3559d61a9b6c37577b6a3076aabfc5238234bf8120f096f52219f6261484e\": container with ID starting with bcf3559d61a9b6c37577b6a3076aabfc5238234bf8120f096f52219f6261484e not found: ID does not exist" containerID="bcf3559d61a9b6c37577b6a3076aabfc5238234bf8120f096f52219f6261484e" Sep 09 10:58:58.565325 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.565276 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcf3559d61a9b6c37577b6a3076aabfc5238234bf8120f096f52219f6261484e"} err="failed to get container status \"bcf3559d61a9b6c37577b6a3076aabfc5238234bf8120f096f52219f6261484e\": rpc error: code = NotFound desc = could not find container \"bcf3559d61a9b6c37577b6a3076aabfc5238234bf8120f096f52219f6261484e\": container with ID starting with bcf3559d61a9b6c37577b6a3076aabfc5238234bf8120f096f52219f6261484e not found: ID does not exist" Sep 09 10:58:58.565325 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.565296 2559 scope.go:117] "RemoveContainer" containerID="80d10926a1959e0f78391d1fa7bf80440ab4f310338b6cdbeac55011fbdaa3bf" Sep 09 10:58:58.573149 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.573121 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f7ff8b-c9d44cdf8-t6t46"] Sep 09 10:58:58.574808 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.574789 2559 scope.go:117] "RemoveContainer" containerID="80d10926a1959e0f78391d1fa7bf80440ab4f310338b6cdbeac55011fbdaa3bf" Sep 09 10:58:58.575148 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:58:58.575124 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80d10926a1959e0f78391d1fa7bf80440ab4f310338b6cdbeac55011fbdaa3bf\": container with ID starting with 80d10926a1959e0f78391d1fa7bf80440ab4f310338b6cdbeac55011fbdaa3bf not found: ID does not exist" containerID="80d10926a1959e0f78391d1fa7bf80440ab4f310338b6cdbeac55011fbdaa3bf" Sep 09 10:58:58.575190 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.575158 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80d10926a1959e0f78391d1fa7bf80440ab4f310338b6cdbeac55011fbdaa3bf"} err="failed to get container status \"80d10926a1959e0f78391d1fa7bf80440ab4f310338b6cdbeac55011fbdaa3bf\": rpc error: code = NotFound desc = could not find container \"80d10926a1959e0f78391d1fa7bf80440ab4f310338b6cdbeac55011fbdaa3bf\": container with ID starting with 80d10926a1959e0f78391d1fa7bf80440ab4f310338b6cdbeac55011fbdaa3bf not found: ID does not exist" Sep 09 10:58:58.578217 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.578188 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f7ff8b-c9d44cdf8-t6t46"] Sep 09 10:58:58.589171 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.589141 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f7ff8b-866bb5656f-gsbkw"] Sep 09 10:58:58.593540 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:58:58.593485 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f7ff8b-866bb5656f-gsbkw"] Sep 09 10:59:00.335481 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:00.335438 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e0f57e3-f642-40b4-bdb8-acc008f311d8" path="/var/lib/kubelet/pods/8e0f57e3-f642-40b4-bdb8-acc008f311d8/volumes" Sep 09 10:59:00.335999 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:00.335980 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d87c55d2-4dbf-49d0-9174-f8ad8220610f" path="/var/lib/kubelet/pods/d87c55d2-4dbf-49d0-9174-f8ad8220610f/volumes" Sep 09 10:59:00.655284 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:00.655202 2559 scope.go:117] "RemoveContainer" containerID="55f5cc71d3d4fa782b9a19c09c39f21242c9a4ffb9b0f76ba95ab1d9b19adcc1" Sep 09 10:59:09.198241 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:09.198207 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33d685-bx42f"] Sep 09 10:59:09.198711 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:09.198614 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e0f57e3-f642-40b4-bdb8-acc008f311d8" containerName="payload-pre-processing" Sep 09 10:59:09.198711 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:09.198627 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e0f57e3-f642-40b4-bdb8-acc008f311d8" containerName="payload-pre-processing" Sep 09 10:59:09.198711 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:09.198636 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f59f4115-1ff3-411f-bfb0-267eb87a7a73" containerName="revoke-keys" Sep 09 10:59:09.198711 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:09.198641 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f59f4115-1ff3-411f-bfb0-267eb87a7a73" containerName="revoke-keys" Sep 09 10:59:09.198711 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:09.198653 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d87c55d2-4dbf-49d0-9174-f8ad8220610f" containerName="payload-processing" Sep 09 10:59:09.198711 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:09.198659 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d87c55d2-4dbf-49d0-9174-f8ad8220610f" containerName="payload-processing" Sep 09 10:59:09.198913 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:09.198726 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f59f4115-1ff3-411f-bfb0-267eb87a7a73" containerName="revoke-keys" Sep 09 10:59:09.198913 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:09.198734 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e0f57e3-f642-40b4-bdb8-acc008f311d8" containerName="payload-pre-processing" Sep 09 10:59:09.198913 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:09.198742 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d87c55d2-4dbf-49d0-9174-f8ad8220610f" containerName="payload-processing" Sep 09 10:59:09.201771 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:09.201755 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33d685-bx42f" Sep 09 10:59:09.204709 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:09.204687 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 10:59:09.205265 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:09.205248 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kq7qv\"" Sep 09 10:59:09.209800 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:09.209775 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33d685-bx42f"] Sep 09 10:59:09.304261 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:09.304228 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a60f17e-9d60-45f6-aaea-f47d77290ff4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-33d685-bx42f\" (UID: \"6a60f17e-9d60-45f6-aaea-f47d77290ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33d685-bx42f" Sep 09 10:59:09.404930 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:09.404892 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a60f17e-9d60-45f6-aaea-f47d77290ff4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-33d685-bx42f\" (UID: \"6a60f17e-9d60-45f6-aaea-f47d77290ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33d685-bx42f" Sep 09 10:59:09.405604 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:09.405583 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a60f17e-9d60-45f6-aaea-f47d77290ff4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-33d685-bx42f\" (UID: \"6a60f17e-9d60-45f6-aaea-f47d77290ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33d685-bx42f" Sep 09 10:59:09.514118 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:09.514034 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33d685-bx42f" Sep 09 10:59:09.786858 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:09.786834 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33d685-bx42f"] Sep 09 10:59:09.788924 ip-10-0-139-133 kubenswrapper[2559]: W0909 10:59:09.788892 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a60f17e_9d60_45f6_aaea_f47d77290ff4.slice/crio-20596f100ce565c5ae37566d56af7d16c253a1dbaec2db65ad269cc2684eafde WatchSource:0}: Error finding container 20596f100ce565c5ae37566d56af7d16c253a1dbaec2db65ad269cc2684eafde: Status 404 returned error can't find the container with id 20596f100ce565c5ae37566d56af7d16c253a1dbaec2db65ad269cc2684eafde Sep 09 10:59:10.624107 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:10.624073 2559 generic.go:356] "Generic (PLEG): container finished" podID="6a60f17e-9d60-45f6-aaea-f47d77290ff4" containerID="ca2d67ca3ebbcc7611508ddb540adcbff88f4dbb4cf269575bc993660c9440c6" exitCode=0 Sep 09 10:59:10.624475 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:10.624126 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33d685-bx42f" event={"ID":"6a60f17e-9d60-45f6-aaea-f47d77290ff4","Type":"ContainerDied","Data":"ca2d67ca3ebbcc7611508ddb540adcbff88f4dbb4cf269575bc993660c9440c6"} Sep 09 10:59:10.624475 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:10.624187 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33d685-bx42f" event={"ID":"6a60f17e-9d60-45f6-aaea-f47d77290ff4","Type":"ContainerStarted","Data":"20596f100ce565c5ae37566d56af7d16c253a1dbaec2db65ad269cc2684eafde"} Sep 09 10:59:11.764697 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:11.764665 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33d685-bx42f" Sep 09 10:59:11.929097 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:11.929011 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a60f17e-9d60-45f6-aaea-f47d77290ff4-maas-api-service-ca\") pod \"6a60f17e-9d60-45f6-aaea-f47d77290ff4\" (UID: \"6a60f17e-9d60-45f6-aaea-f47d77290ff4\") " Sep 09 10:59:11.929365 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:11.929338 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a60f17e-9d60-45f6-aaea-f47d77290ff4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6a60f17e-9d60-45f6-aaea-f47d77290ff4" (UID: "6a60f17e-9d60-45f6-aaea-f47d77290ff4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:59:12.029935 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:12.029902 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a60f17e-9d60-45f6-aaea-f47d77290ff4-maas-api-service-ca\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:59:12.635452 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:12.635364 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33d685-bx42f" event={"ID":"6a60f17e-9d60-45f6-aaea-f47d77290ff4","Type":"ContainerDied","Data":"20596f100ce565c5ae37566d56af7d16c253a1dbaec2db65ad269cc2684eafde"} Sep 09 10:59:12.635452 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:12.635388 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33d685-bx42f" Sep 09 10:59:12.635452 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:12.635413 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20596f100ce565c5ae37566d56af7d16c253a1dbaec2db65ad269cc2684eafde" Sep 09 10:59:25.091483 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:25.091447 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-33d685-7c7cd7c479-rbhzn"] Sep 09 10:59:25.091951 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:25.091665 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-33d685-7c7cd7c479-rbhzn" podUID="60aea43a-6bf8-406d-80e2-61b46e25c98f" containerName="payload-processing" containerID="cri-o://705fb2eff2472b74bf77b4276cf8e06250045056f426c094708763699f60fd81" gracePeriod=30 Sep 09 10:59:25.332124 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:25.332093 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33d685-bx42f"] Sep 09 10:59:25.338775 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:25.338752 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33d685-bx42f"] Sep 09 10:59:25.345977 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:25.345900 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-33d685-7c7cd7c479-rbhzn" Sep 09 10:59:25.447044 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:25.447008 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60aea43a-6bf8-406d-80e2-61b46e25c98f-plugins-config-volume\") pod \"60aea43a-6bf8-406d-80e2-61b46e25c98f\" (UID: \"60aea43a-6bf8-406d-80e2-61b46e25c98f\") " Sep 09 10:59:25.447227 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:25.447130 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h54lf\" (UniqueName: \"kubernetes.io/projected/60aea43a-6bf8-406d-80e2-61b46e25c98f-kube-api-access-h54lf\") pod \"60aea43a-6bf8-406d-80e2-61b46e25c98f\" (UID: \"60aea43a-6bf8-406d-80e2-61b46e25c98f\") " Sep 09 10:59:25.447432 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:25.447406 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60aea43a-6bf8-406d-80e2-61b46e25c98f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "60aea43a-6bf8-406d-80e2-61b46e25c98f" (UID: "60aea43a-6bf8-406d-80e2-61b46e25c98f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:59:25.449217 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:25.449189 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60aea43a-6bf8-406d-80e2-61b46e25c98f-kube-api-access-h54lf" (OuterVolumeSpecName: "kube-api-access-h54lf") pod "60aea43a-6bf8-406d-80e2-61b46e25c98f" (UID: "60aea43a-6bf8-406d-80e2-61b46e25c98f"). InnerVolumeSpecName "kube-api-access-h54lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:59:25.548004 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:25.547970 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h54lf\" (UniqueName: \"kubernetes.io/projected/60aea43a-6bf8-406d-80e2-61b46e25c98f-kube-api-access-h54lf\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:59:25.548004 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:25.547998 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60aea43a-6bf8-406d-80e2-61b46e25c98f-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 10:59:25.710957 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:25.710881 2559 generic.go:356] "Generic (PLEG): container finished" podID="60aea43a-6bf8-406d-80e2-61b46e25c98f" containerID="705fb2eff2472b74bf77b4276cf8e06250045056f426c094708763699f60fd81" exitCode=0 Sep 09 10:59:25.710957 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:25.710921 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-33d685-7c7cd7c479-rbhzn" Sep 09 10:59:25.710957 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:25.710927 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-33d685-7c7cd7c479-rbhzn" event={"ID":"60aea43a-6bf8-406d-80e2-61b46e25c98f","Type":"ContainerDied","Data":"705fb2eff2472b74bf77b4276cf8e06250045056f426c094708763699f60fd81"} Sep 09 10:59:25.711182 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:25.710969 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-33d685-7c7cd7c479-rbhzn" event={"ID":"60aea43a-6bf8-406d-80e2-61b46e25c98f","Type":"ContainerDied","Data":"a9a0073176513f91b0fd932950857f8c86341e815a56265cff004632bc3ca2aa"} Sep 09 10:59:25.711182 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:25.710992 2559 scope.go:117] "RemoveContainer" containerID="705fb2eff2472b74bf77b4276cf8e06250045056f426c094708763699f60fd81" Sep 09 10:59:25.720368 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:25.720351 2559 scope.go:117] "RemoveContainer" containerID="705fb2eff2472b74bf77b4276cf8e06250045056f426c094708763699f60fd81" Sep 09 10:59:25.720655 ip-10-0-139-133 kubenswrapper[2559]: E0909 10:59:25.720636 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"705fb2eff2472b74bf77b4276cf8e06250045056f426c094708763699f60fd81\": container with ID starting with 705fb2eff2472b74bf77b4276cf8e06250045056f426c094708763699f60fd81 not found: ID does not exist" containerID="705fb2eff2472b74bf77b4276cf8e06250045056f426c094708763699f60fd81" Sep 09 10:59:25.720718 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:25.720665 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"705fb2eff2472b74bf77b4276cf8e06250045056f426c094708763699f60fd81"} err="failed to get container status \"705fb2eff2472b74bf77b4276cf8e06250045056f426c094708763699f60fd81\": rpc error: code = NotFound desc = could not find container \"705fb2eff2472b74bf77b4276cf8e06250045056f426c094708763699f60fd81\": container with ID starting with 705fb2eff2472b74bf77b4276cf8e06250045056f426c094708763699f60fd81 not found: ID does not exist" Sep 09 10:59:25.737167 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:25.737145 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-33d685-7c7cd7c479-rbhzn"] Sep 09 10:59:25.739025 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:25.738995 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-33d685-7c7cd7c479-rbhzn"] Sep 09 10:59:26.334282 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:26.334252 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60aea43a-6bf8-406d-80e2-61b46e25c98f" path="/var/lib/kubelet/pods/60aea43a-6bf8-406d-80e2-61b46e25c98f/volumes" Sep 09 10:59:26.334676 ip-10-0-139-133 kubenswrapper[2559]: I0909 10:59:26.334587 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a60f17e-9d60-45f6-aaea-f47d77290ff4" path="/var/lib/kubelet/pods/6a60f17e-9d60-45f6-aaea-f47d77290ff4/volumes" Sep 09 11:00:00.145590 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:00.145553 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c1bfbb-29815860-hcfdn"] Sep 09 11:00:00.146008 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:00.145964 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a60f17e-9d60-45f6-aaea-f47d77290ff4" containerName="revoke-keys" Sep 09 11:00:00.146008 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:00.145978 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a60f17e-9d60-45f6-aaea-f47d77290ff4" containerName="revoke-keys" Sep 09 11:00:00.146008 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:00.145993 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60aea43a-6bf8-406d-80e2-61b46e25c98f" containerName="payload-processing" Sep 09 11:00:00.146008 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:00.145999 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="60aea43a-6bf8-406d-80e2-61b46e25c98f" containerName="payload-processing" Sep 09 11:00:00.146138 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:00.146065 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a60f17e-9d60-45f6-aaea-f47d77290ff4" containerName="revoke-keys" Sep 09 11:00:00.146138 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:00.146077 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="60aea43a-6bf8-406d-80e2-61b46e25c98f" containerName="payload-processing" Sep 09 11:00:00.152139 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:00.152102 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815860-k5dlj"] Sep 09 11:00:00.153906 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:00.153878 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c1bfbb-29815860-hcfdn" Sep 09 11:00:00.156830 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:00.156797 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-535b75-29815860-v2267"] Sep 09 11:00:00.157311 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:00.157287 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815860-k5dlj" Sep 09 11:00:00.157736 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:00.157700 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kq7qv\"" Sep 09 11:00:00.160775 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:00.160744 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c1bfbb-29815860-hcfdn"] Sep 09 11:00:00.160884 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:00.160828 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-535b75-29815860-v2267" Sep 09 11:00:00.166734 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:00.166701 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-535b75-29815860-v2267"] Sep 09 11:00:00.168041 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:00.168019 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c1bfbb-29815860-hcfdn" Sep 09 11:00:00.180157 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:00.180129 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815860-k5dlj"] Sep 09 11:00:00.181300 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:00.181278 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815860-k5dlj" Sep 09 11:00:00.217251 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:00.217222 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-535b75-29815860-v2267" Sep 09 11:00:00.450166 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:00.450141 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c1bfbb-29815860-hcfdn"] Sep 09 11:00:00.453026 ip-10-0-139-133 kubenswrapper[2559]: W0909 11:00:00.453001 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ad0bfd_9982_44f5_8c12_c3141a3c9a7e.slice/crio-8c5b38cc72715999e194e708ea2a1509ae32c3dbad0ac44f2ca5f7260fea4fa5 WatchSource:0}: Error finding container 8c5b38cc72715999e194e708ea2a1509ae32c3dbad0ac44f2ca5f7260fea4fa5: Status 404 returned error can't find the container with id 8c5b38cc72715999e194e708ea2a1509ae32c3dbad0ac44f2ca5f7260fea4fa5 Sep 09 11:00:00.476236 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:00.476202 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815860-k5dlj"] Sep 09 11:00:00.482075 ip-10-0-139-133 kubenswrapper[2559]: W0909 11:00:00.482050 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55468590_26e0_40b1_92db_dd92657ebd47.slice/crio-b2762c773afb902e685ecc9cdc919230211f8b99d863286ba84a0e7f7e71049e WatchSource:0}: Error finding container b2762c773afb902e685ecc9cdc919230211f8b99d863286ba84a0e7f7e71049e: Status 404 returned error can't find the container with id b2762c773afb902e685ecc9cdc919230211f8b99d863286ba84a0e7f7e71049e Sep 09 11:00:00.506198 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:00.506174 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-535b75-29815860-v2267"] Sep 09 11:00:00.515664 ip-10-0-139-133 kubenswrapper[2559]: W0909 11:00:00.515638 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod000569a5_038c_421d_b725_35765e669175.slice/crio-e1263605921334370cfcb1a45879061e952df09a9b2d30e28c56dcf427cec2b7 WatchSource:0}: Error finding container e1263605921334370cfcb1a45879061e952df09a9b2d30e28c56dcf427cec2b7: Status 404 returned error can't find the container with id e1263605921334370cfcb1a45879061e952df09a9b2d30e28c56dcf427cec2b7 Sep 09 11:00:00.899146 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:00.899110 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-535b75-29815860-v2267" event={"ID":"000569a5-038c-421d-b725-35765e669175","Type":"ContainerStarted","Data":"3fb6b2fcf39ff2b988018f6858283f3fa4b04b4c794741a7149d42b008de74f6"} Sep 09 11:00:00.899365 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:00.899165 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-535b75-29815860-v2267" event={"ID":"000569a5-038c-421d-b725-35765e669175","Type":"ContainerStarted","Data":"e1263605921334370cfcb1a45879061e952df09a9b2d30e28c56dcf427cec2b7"} Sep 09 11:00:00.900707 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:00.900682 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815860-k5dlj" event={"ID":"55468590-26e0-40b1-92db-dd92657ebd47","Type":"ContainerStarted","Data":"215de9b3a7b38fc28db730e43f03e30b0a34f93f9b3523fc7eb7c9143cffddac"} Sep 09 11:00:00.900846 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:00.900726 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815860-k5dlj" event={"ID":"55468590-26e0-40b1-92db-dd92657ebd47","Type":"ContainerStarted","Data":"b2762c773afb902e685ecc9cdc919230211f8b99d863286ba84a0e7f7e71049e"} Sep 09 11:00:00.902261 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:00.902233 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c1bfbb-29815860-hcfdn" event={"ID":"67ad0bfd-9982-44f5-8c12-c3141a3c9a7e","Type":"ContainerStarted","Data":"56e4cd032396a0a6aadc1993310f84efa5be27456f47f2181764c8bdb4ddcd73"} Sep 09 11:00:00.902367 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:00.902288 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c1bfbb-29815860-hcfdn" event={"ID":"67ad0bfd-9982-44f5-8c12-c3141a3c9a7e","Type":"ContainerStarted","Data":"8c5b38cc72715999e194e708ea2a1509ae32c3dbad0ac44f2ca5f7260fea4fa5"} Sep 09 11:00:00.914709 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:00.914662 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-535b75-29815860-v2267" podStartSLOduration=0.914649237 podStartE2EDuration="914.649237ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 11:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 11:00:00.91336293 +0000 UTC m=+2041.087965253" watchObservedRunningTime="2026-09-09 11:00:00.914649237 +0000 UTC m=+2041.089251562" Sep 09 11:00:00.930927 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:00.930871 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c1bfbb-29815860-hcfdn" podStartSLOduration=0.930852274 podStartE2EDuration="930.852274ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 11:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 11:00:00.928760299 +0000 UTC m=+2041.103362624" watchObservedRunningTime="2026-09-09 11:00:00.930852274 +0000 UTC m=+2041.105454598" Sep 09 11:00:00.944596 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:00.944540 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815860-k5dlj" podStartSLOduration=0.944525831 podStartE2EDuration="944.525831ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 11:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 11:00:00.943301336 +0000 UTC m=+2041.117903663" watchObservedRunningTime="2026-09-09 11:00:00.944525831 +0000 UTC m=+2041.119128151" Sep 09 11:00:01.909303 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:01.909264 2559 generic.go:356] "Generic (PLEG): container finished" podID="000569a5-038c-421d-b725-35765e669175" containerID="3fb6b2fcf39ff2b988018f6858283f3fa4b04b4c794741a7149d42b008de74f6" exitCode=0 Sep 09 11:00:01.909771 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:01.909344 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-535b75-29815860-v2267" event={"ID":"000569a5-038c-421d-b725-35765e669175","Type":"ContainerDied","Data":"3fb6b2fcf39ff2b988018f6858283f3fa4b04b4c794741a7149d42b008de74f6"} Sep 09 11:00:01.910807 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:01.910787 2559 generic.go:356] "Generic (PLEG): container finished" podID="55468590-26e0-40b1-92db-dd92657ebd47" containerID="215de9b3a7b38fc28db730e43f03e30b0a34f93f9b3523fc7eb7c9143cffddac" exitCode=0 Sep 09 11:00:01.910918 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:01.910813 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815860-k5dlj" event={"ID":"55468590-26e0-40b1-92db-dd92657ebd47","Type":"ContainerDied","Data":"215de9b3a7b38fc28db730e43f03e30b0a34f93f9b3523fc7eb7c9143cffddac"} Sep 09 11:00:01.912227 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:01.912202 2559 generic.go:356] "Generic (PLEG): container finished" podID="67ad0bfd-9982-44f5-8c12-c3141a3c9a7e" containerID="56e4cd032396a0a6aadc1993310f84efa5be27456f47f2181764c8bdb4ddcd73" exitCode=0 Sep 09 11:00:01.912339 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:01.912263 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c1bfbb-29815860-hcfdn" event={"ID":"67ad0bfd-9982-44f5-8c12-c3141a3c9a7e","Type":"ContainerDied","Data":"56e4cd032396a0a6aadc1993310f84efa5be27456f47f2181764c8bdb4ddcd73"} Sep 09 11:00:03.052343 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:03.052314 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c1bfbb-29815860-hcfdn" Sep 09 11:00:03.101732 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:03.101703 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-535b75-29815860-v2267" Sep 09 11:00:03.131066 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:03.131038 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815860-k5dlj" Sep 09 11:00:03.923824 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:03.923792 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-535b75-29815860-v2267" Sep 09 11:00:03.924048 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:03.923792 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-535b75-29815860-v2267" event={"ID":"000569a5-038c-421d-b725-35765e669175","Type":"ContainerDied","Data":"e1263605921334370cfcb1a45879061e952df09a9b2d30e28c56dcf427cec2b7"} Sep 09 11:00:03.924048 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:03.923947 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1263605921334370cfcb1a45879061e952df09a9b2d30e28c56dcf427cec2b7" Sep 09 11:00:03.925235 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:03.925211 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815860-k5dlj" Sep 09 11:00:03.925372 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:03.925256 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815860-k5dlj" event={"ID":"55468590-26e0-40b1-92db-dd92657ebd47","Type":"ContainerDied","Data":"b2762c773afb902e685ecc9cdc919230211f8b99d863286ba84a0e7f7e71049e"} Sep 09 11:00:03.925372 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:03.925309 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2762c773afb902e685ecc9cdc919230211f8b99d863286ba84a0e7f7e71049e" Sep 09 11:00:03.926669 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:03.926652 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c1bfbb-29815860-hcfdn" Sep 09 11:00:03.926793 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:03.926687 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c1bfbb-29815860-hcfdn" event={"ID":"67ad0bfd-9982-44f5-8c12-c3141a3c9a7e","Type":"ContainerDied","Data":"8c5b38cc72715999e194e708ea2a1509ae32c3dbad0ac44f2ca5f7260fea4fa5"} Sep 09 11:00:03.926793 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:03.926726 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c5b38cc72715999e194e708ea2a1509ae32c3dbad0ac44f2ca5f7260fea4fa5" Sep 09 11:00:15.129138 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:15.129099 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1bfbb-vg5wx"] Sep 09 11:00:15.129710 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:15.129591 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67ad0bfd-9982-44f5-8c12-c3141a3c9a7e" containerName="cleanup" Sep 09 11:00:15.129710 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:15.129606 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="67ad0bfd-9982-44f5-8c12-c3141a3c9a7e" containerName="cleanup" Sep 09 11:00:15.129710 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:15.129626 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="000569a5-038c-421d-b725-35765e669175" containerName="cleanup" Sep 09 11:00:15.129710 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:15.129632 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="000569a5-038c-421d-b725-35765e669175" containerName="cleanup" Sep 09 11:00:15.129710 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:15.129648 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55468590-26e0-40b1-92db-dd92657ebd47" containerName="cleanup" Sep 09 11:00:15.129710 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:15.129656 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="55468590-26e0-40b1-92db-dd92657ebd47" containerName="cleanup" Sep 09 11:00:15.129938 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:15.129717 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="000569a5-038c-421d-b725-35765e669175" containerName="cleanup" Sep 09 11:00:15.129938 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:15.129726 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="67ad0bfd-9982-44f5-8c12-c3141a3c9a7e" containerName="cleanup" Sep 09 11:00:15.129938 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:15.129734 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="55468590-26e0-40b1-92db-dd92657ebd47" containerName="cleanup" Sep 09 11:00:15.132835 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:15.132817 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1bfbb-vg5wx" Sep 09 11:00:15.135600 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:15.135577 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 11:00:15.135718 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:15.135654 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kq7qv\"" Sep 09 11:00:15.140335 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:15.140304 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1bfbb-vg5wx"] Sep 09 11:00:15.186068 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:15.186026 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6705f7ee-06c2-4c00-aa06-89c1a83a3715-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c1bfbb-vg5wx\" (UID: \"6705f7ee-06c2-4c00-aa06-89c1a83a3715\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1bfbb-vg5wx" Sep 09 11:00:15.286699 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:15.286662 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6705f7ee-06c2-4c00-aa06-89c1a83a3715-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c1bfbb-vg5wx\" (UID: \"6705f7ee-06c2-4c00-aa06-89c1a83a3715\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1bfbb-vg5wx" Sep 09 11:00:15.287280 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:15.287259 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6705f7ee-06c2-4c00-aa06-89c1a83a3715-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c1bfbb-vg5wx\" (UID: \"6705f7ee-06c2-4c00-aa06-89c1a83a3715\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1bfbb-vg5wx" Sep 09 11:00:15.445319 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:15.445215 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1bfbb-vg5wx" Sep 09 11:00:15.724425 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:15.724398 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1bfbb-vg5wx"] Sep 09 11:00:15.726331 ip-10-0-139-133 kubenswrapper[2559]: W0909 11:00:15.726300 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6705f7ee_06c2_4c00_aa06_89c1a83a3715.slice/crio-daa6fc3ab1207414552a6ad2c486c11303d27c03956dcdbe99030b016d018be0 WatchSource:0}: Error finding container daa6fc3ab1207414552a6ad2c486c11303d27c03956dcdbe99030b016d018be0: Status 404 returned error can't find the container with id daa6fc3ab1207414552a6ad2c486c11303d27c03956dcdbe99030b016d018be0 Sep 09 11:00:15.993151 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:15.993061 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1bfbb-vg5wx" event={"ID":"6705f7ee-06c2-4c00-aa06-89c1a83a3715","Type":"ContainerStarted","Data":"daa6fc3ab1207414552a6ad2c486c11303d27c03956dcdbe99030b016d018be0"} Sep 09 11:00:17.000778 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:17.000744 2559 generic.go:356] "Generic (PLEG): container finished" podID="6705f7ee-06c2-4c00-aa06-89c1a83a3715" containerID="e2ed2c48ea68e545a32eab95aab587b8b676ba13ea681a68468f6f95ad3043ad" exitCode=0 Sep 09 11:00:17.001213 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:17.000829 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1bfbb-vg5wx" event={"ID":"6705f7ee-06c2-4c00-aa06-89c1a83a3715","Type":"ContainerDied","Data":"e2ed2c48ea68e545a32eab95aab587b8b676ba13ea681a68468f6f95ad3043ad"} Sep 09 11:00:18.139342 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:18.139314 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1bfbb-vg5wx" Sep 09 11:00:18.212166 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:18.212123 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6705f7ee-06c2-4c00-aa06-89c1a83a3715-maas-api-service-ca\") pod \"6705f7ee-06c2-4c00-aa06-89c1a83a3715\" (UID: \"6705f7ee-06c2-4c00-aa06-89c1a83a3715\") " Sep 09 11:00:18.212492 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:18.212468 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6705f7ee-06c2-4c00-aa06-89c1a83a3715-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6705f7ee-06c2-4c00-aa06-89c1a83a3715" (UID: "6705f7ee-06c2-4c00-aa06-89c1a83a3715"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 11:00:18.313326 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:18.313233 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6705f7ee-06c2-4c00-aa06-89c1a83a3715-maas-api-service-ca\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 11:00:19.012589 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:19.012549 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1bfbb-vg5wx" event={"ID":"6705f7ee-06c2-4c00-aa06-89c1a83a3715","Type":"ContainerDied","Data":"daa6fc3ab1207414552a6ad2c486c11303d27c03956dcdbe99030b016d018be0"} Sep 09 11:00:19.012589 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:19.012568 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1bfbb-vg5wx" Sep 09 11:00:19.012810 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:19.012606 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="daa6fc3ab1207414552a6ad2c486c11303d27c03956dcdbe99030b016d018be0" Sep 09 11:00:25.755958 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:25.755882 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c1bfbb-29815860-hcfdn"] Sep 09 11:00:25.764858 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:25.764828 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c1bfbb-29815860-hcfdn"] Sep 09 11:00:25.806357 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:25.806316 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz"] Sep 09 11:00:25.806579 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:25.806556 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz" podUID="7cd21fb4-40d9-4d30-9b04-9ed81a2b540e" containerName="payload-processing" containerID="cri-o://11b51c4d3c9a39682e00ab3516b48709336b3e87affa735d93a89a1df7cbe168" gracePeriod=30 Sep 09 11:00:25.816388 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:25.816354 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c1bfbb-f47f7b7bb-t69bz"] Sep 09 11:00:25.816631 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:25.816584 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1bfbb-f47f7b7bb-t69bz" podUID="775ee7e4-76f8-4cb1-af48-f8b82716110a" containerName="payload-pre-processing" containerID="cri-o://fee9d50c70b4fe6026b1508c9b6ea6d001e18989e207a10b5a240cc6e16186ca" gracePeriod=30 Sep 09 11:00:26.050808 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:26.050761 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1bfbb-vg5wx"] Sep 09 11:00:26.053757 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:26.053707 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1bfbb-vg5wx"] Sep 09 11:00:26.056427 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:26.056394 2559 generic.go:356] "Generic (PLEG): container finished" podID="7cd21fb4-40d9-4d30-9b04-9ed81a2b540e" containerID="11b51c4d3c9a39682e00ab3516b48709336b3e87affa735d93a89a1df7cbe168" exitCode=0 Sep 09 11:00:26.056627 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:26.056594 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz" event={"ID":"7cd21fb4-40d9-4d30-9b04-9ed81a2b540e","Type":"ContainerDied","Data":"11b51c4d3c9a39682e00ab3516b48709336b3e87affa735d93a89a1df7cbe168"} Sep 09 11:00:26.058589 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:26.058489 2559 generic.go:356] "Generic (PLEG): container finished" podID="775ee7e4-76f8-4cb1-af48-f8b82716110a" containerID="fee9d50c70b4fe6026b1508c9b6ea6d001e18989e207a10b5a240cc6e16186ca" exitCode=0 Sep 09 11:00:26.058589 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:26.058540 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1bfbb-f47f7b7bb-t69bz" event={"ID":"775ee7e4-76f8-4cb1-af48-f8b82716110a","Type":"ContainerDied","Data":"fee9d50c70b4fe6026b1508c9b6ea6d001e18989e207a10b5a240cc6e16186ca"} Sep 09 11:00:26.079416 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:26.079389 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz" Sep 09 11:00:26.106878 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:26.106851 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1bfbb-f47f7b7bb-t69bz" Sep 09 11:00:26.186563 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:26.186524 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/775ee7e4-76f8-4cb1-af48-f8b82716110a-plugins-config-volume\") pod \"775ee7e4-76f8-4cb1-af48-f8b82716110a\" (UID: \"775ee7e4-76f8-4cb1-af48-f8b82716110a\") " Sep 09 11:00:26.186766 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:26.186581 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd21fb4-40d9-4d30-9b04-9ed81a2b540e-plugins-config-volume\") pod \"7cd21fb4-40d9-4d30-9b04-9ed81a2b540e\" (UID: \"7cd21fb4-40d9-4d30-9b04-9ed81a2b540e\") " Sep 09 11:00:26.186766 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:26.186609 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltwmh\" (UniqueName: \"kubernetes.io/projected/775ee7e4-76f8-4cb1-af48-f8b82716110a-kube-api-access-ltwmh\") pod \"775ee7e4-76f8-4cb1-af48-f8b82716110a\" (UID: \"775ee7e4-76f8-4cb1-af48-f8b82716110a\") " Sep 09 11:00:26.186766 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:26.186726 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6qlb\" (UniqueName: \"kubernetes.io/projected/7cd21fb4-40d9-4d30-9b04-9ed81a2b540e-kube-api-access-k6qlb\") pod \"7cd21fb4-40d9-4d30-9b04-9ed81a2b540e\" (UID: \"7cd21fb4-40d9-4d30-9b04-9ed81a2b540e\") " Sep 09 11:00:26.187027 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:26.186994 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cd21fb4-40d9-4d30-9b04-9ed81a2b540e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7cd21fb4-40d9-4d30-9b04-9ed81a2b540e" (UID: "7cd21fb4-40d9-4d30-9b04-9ed81a2b540e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 11:00:26.187126 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:26.186995 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/775ee7e4-76f8-4cb1-af48-f8b82716110a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "775ee7e4-76f8-4cb1-af48-f8b82716110a" (UID: "775ee7e4-76f8-4cb1-af48-f8b82716110a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 11:00:26.188976 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:26.188952 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd21fb4-40d9-4d30-9b04-9ed81a2b540e-kube-api-access-k6qlb" (OuterVolumeSpecName: "kube-api-access-k6qlb") pod "7cd21fb4-40d9-4d30-9b04-9ed81a2b540e" (UID: "7cd21fb4-40d9-4d30-9b04-9ed81a2b540e"). InnerVolumeSpecName "kube-api-access-k6qlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 11:00:26.189072 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:26.188989 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/775ee7e4-76f8-4cb1-af48-f8b82716110a-kube-api-access-ltwmh" (OuterVolumeSpecName: "kube-api-access-ltwmh") pod "775ee7e4-76f8-4cb1-af48-f8b82716110a" (UID: "775ee7e4-76f8-4cb1-af48-f8b82716110a"). InnerVolumeSpecName "kube-api-access-ltwmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 11:00:26.288285 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:26.288199 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltwmh\" (UniqueName: \"kubernetes.io/projected/775ee7e4-76f8-4cb1-af48-f8b82716110a-kube-api-access-ltwmh\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 11:00:26.288285 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:26.288231 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6qlb\" (UniqueName: \"kubernetes.io/projected/7cd21fb4-40d9-4d30-9b04-9ed81a2b540e-kube-api-access-k6qlb\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 11:00:26.288285 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:26.288240 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/775ee7e4-76f8-4cb1-af48-f8b82716110a-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 11:00:26.288285 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:26.288249 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd21fb4-40d9-4d30-9b04-9ed81a2b540e-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 11:00:26.334834 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:26.334799 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6705f7ee-06c2-4c00-aa06-89c1a83a3715" path="/var/lib/kubelet/pods/6705f7ee-06c2-4c00-aa06-89c1a83a3715/volumes" Sep 09 11:00:26.335082 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:26.335069 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67ad0bfd-9982-44f5-8c12-c3141a3c9a7e" path="/var/lib/kubelet/pods/67ad0bfd-9982-44f5-8c12-c3141a3c9a7e/volumes" Sep 09 11:00:27.067356 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:27.067320 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1bfbb-f47f7b7bb-t69bz" Sep 09 11:00:27.067356 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:27.067340 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c1bfbb-f47f7b7bb-t69bz" event={"ID":"775ee7e4-76f8-4cb1-af48-f8b82716110a","Type":"ContainerDied","Data":"0864d92f41b27d315820f989c2ebbc9cfbf0a6a5d5aef41967be957a6216e830"} Sep 09 11:00:27.067923 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:27.067402 2559 scope.go:117] "RemoveContainer" containerID="fee9d50c70b4fe6026b1508c9b6ea6d001e18989e207a10b5a240cc6e16186ca" Sep 09 11:00:27.069081 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:27.069056 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz" event={"ID":"7cd21fb4-40d9-4d30-9b04-9ed81a2b540e","Type":"ContainerDied","Data":"a9f04ba9f8a679f55669d032dae2f34045f5d481357d14192ce4a4ccb0747cc0"} Sep 09 11:00:27.069211 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:27.069059 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz" Sep 09 11:00:27.077331 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:27.077308 2559 scope.go:117] "RemoveContainer" containerID="11b51c4d3c9a39682e00ab3516b48709336b3e87affa735d93a89a1df7cbe168" Sep 09 11:00:27.089005 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:27.088974 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz"] Sep 09 11:00:27.094312 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:27.094286 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c1bfbb-67684776c-99clz"] Sep 09 11:00:27.104901 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:27.104870 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c1bfbb-f47f7b7bb-t69bz"] Sep 09 11:00:27.112662 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:27.112633 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c1bfbb-f47f7b7bb-t69bz"] Sep 09 11:00:28.336140 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:28.335524 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="775ee7e4-76f8-4cb1-af48-f8b82716110a" path="/var/lib/kubelet/pods/775ee7e4-76f8-4cb1-af48-f8b82716110a/volumes" Sep 09 11:00:28.336140 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:28.336021 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cd21fb4-40d9-4d30-9b04-9ed81a2b540e" path="/var/lib/kubelet/pods/7cd21fb4-40d9-4d30-9b04-9ed81a2b540e/volumes" Sep 09 11:00:33.391783 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:33.391747 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1eb21d-k4qrs"] Sep 09 11:00:33.392178 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:33.392148 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cd21fb4-40d9-4d30-9b04-9ed81a2b540e" containerName="payload-processing" Sep 09 11:00:33.392178 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:33.392164 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cd21fb4-40d9-4d30-9b04-9ed81a2b540e" containerName="payload-processing" Sep 09 11:00:33.392273 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:33.392184 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6705f7ee-06c2-4c00-aa06-89c1a83a3715" containerName="revoke-keys" Sep 09 11:00:33.392273 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:33.392192 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6705f7ee-06c2-4c00-aa06-89c1a83a3715" containerName="revoke-keys" Sep 09 11:00:33.392273 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:33.392219 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="775ee7e4-76f8-4cb1-af48-f8b82716110a" containerName="payload-pre-processing" Sep 09 11:00:33.392273 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:33.392227 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="775ee7e4-76f8-4cb1-af48-f8b82716110a" containerName="payload-pre-processing" Sep 09 11:00:33.392395 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:33.392335 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6705f7ee-06c2-4c00-aa06-89c1a83a3715" containerName="revoke-keys" Sep 09 11:00:33.392395 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:33.392347 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="775ee7e4-76f8-4cb1-af48-f8b82716110a" containerName="payload-pre-processing" Sep 09 11:00:33.392395 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:33.392358 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cd21fb4-40d9-4d30-9b04-9ed81a2b540e" containerName="payload-processing" Sep 09 11:00:33.396991 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:33.396966 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1eb21d-k4qrs" Sep 09 11:00:33.401089 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:33.401065 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 11:00:33.401237 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:33.401065 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kq7qv\"" Sep 09 11:00:33.406425 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:33.406393 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1eb21d-k4qrs"] Sep 09 11:00:33.448609 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:33.448564 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4bb16b3-8cef-4b5f-b215-7edcb11629d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1eb21d-k4qrs\" (UID: \"a4bb16b3-8cef-4b5f-b215-7edcb11629d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1eb21d-k4qrs" Sep 09 11:00:33.549947 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:33.549906 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4bb16b3-8cef-4b5f-b215-7edcb11629d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1eb21d-k4qrs\" (UID: \"a4bb16b3-8cef-4b5f-b215-7edcb11629d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1eb21d-k4qrs" Sep 09 11:00:33.550662 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:33.550639 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4bb16b3-8cef-4b5f-b215-7edcb11629d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1eb21d-k4qrs\" (UID: \"a4bb16b3-8cef-4b5f-b215-7edcb11629d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1eb21d-k4qrs" Sep 09 11:00:33.709955 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:33.709859 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1eb21d-k4qrs" Sep 09 11:00:33.988458 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:33.988428 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1eb21d-k4qrs"] Sep 09 11:00:33.990939 ip-10-0-139-133 kubenswrapper[2559]: W0909 11:00:33.990915 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4bb16b3_8cef_4b5f_b215_7edcb11629d8.slice/crio-8397d846a216136b7eaa25c63e2955e63f95886f41ba17a57eb71a977828416c WatchSource:0}: Error finding container 8397d846a216136b7eaa25c63e2955e63f95886f41ba17a57eb71a977828416c: Status 404 returned error can't find the container with id 8397d846a216136b7eaa25c63e2955e63f95886f41ba17a57eb71a977828416c Sep 09 11:00:34.120880 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:34.120842 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1eb21d-k4qrs" event={"ID":"a4bb16b3-8cef-4b5f-b215-7edcb11629d8","Type":"ContainerStarted","Data":"8397d846a216136b7eaa25c63e2955e63f95886f41ba17a57eb71a977828416c"} Sep 09 11:00:35.128197 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:35.128155 2559 generic.go:356] "Generic (PLEG): container finished" podID="a4bb16b3-8cef-4b5f-b215-7edcb11629d8" containerID="72c4ab9c71f291ac7588542979102fc2e546a7ac6b262ac27610664959256170" exitCode=0 Sep 09 11:00:35.128592 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:35.128236 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1eb21d-k4qrs" event={"ID":"a4bb16b3-8cef-4b5f-b215-7edcb11629d8","Type":"ContainerDied","Data":"72c4ab9c71f291ac7588542979102fc2e546a7ac6b262ac27610664959256170"} Sep 09 11:00:36.270884 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:36.270859 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1eb21d-k4qrs" Sep 09 11:00:36.375466 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:36.375435 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4bb16b3-8cef-4b5f-b215-7edcb11629d8-maas-api-service-ca\") pod \"a4bb16b3-8cef-4b5f-b215-7edcb11629d8\" (UID: \"a4bb16b3-8cef-4b5f-b215-7edcb11629d8\") " Sep 09 11:00:36.375807 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:36.375781 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4bb16b3-8cef-4b5f-b215-7edcb11629d8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a4bb16b3-8cef-4b5f-b215-7edcb11629d8" (UID: "a4bb16b3-8cef-4b5f-b215-7edcb11629d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 11:00:36.476935 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:36.476856 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4bb16b3-8cef-4b5f-b215-7edcb11629d8-maas-api-service-ca\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 11:00:37.139520 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:37.139473 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1eb21d-k4qrs" Sep 09 11:00:37.139702 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:37.139529 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1eb21d-k4qrs" event={"ID":"a4bb16b3-8cef-4b5f-b215-7edcb11629d8","Type":"ContainerDied","Data":"8397d846a216136b7eaa25c63e2955e63f95886f41ba17a57eb71a977828416c"} Sep 09 11:00:37.139702 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:37.139573 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8397d846a216136b7eaa25c63e2955e63f95886f41ba17a57eb71a977828416c" Sep 09 11:00:43.911906 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:43.911870 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1eb21d-57fcdc7b5-n5fr8"] Sep 09 11:00:43.912314 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:43.912135 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1eb21d-57fcdc7b5-n5fr8" podUID="39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d" containerName="payload-processing" containerID="cri-o://36bcc275cfa032b9507ec4d79f5daf100e1b86122072a7be3af9b82ef4d538ea" gracePeriod=30 Sep 09 11:00:43.914887 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:43.914861 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1eb21d-846bcbb58-p64nc"] Sep 09 11:00:43.915076 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:43.915054 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1eb21d-846bcbb58-p64nc" podUID="fc30260f-79d2-4307-ac41-0482974ad1e8" containerName="payload-pre-processing" containerID="cri-o://abe3eb841370db13b923bad64284bcccef8b99ee7c3c26cee9693aec0e34c2e6" gracePeriod=30 Sep 09 11:00:44.085138 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:44.085037 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1eb21d-k4qrs"] Sep 09 11:00:44.088575 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:44.088543 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1eb21d-k4qrs"] Sep 09 11:00:44.179491 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:44.179407 2559 generic.go:356] "Generic (PLEG): container finished" podID="fc30260f-79d2-4307-ac41-0482974ad1e8" containerID="abe3eb841370db13b923bad64284bcccef8b99ee7c3c26cee9693aec0e34c2e6" exitCode=0 Sep 09 11:00:44.179661 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:44.179484 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1eb21d-846bcbb58-p64nc" event={"ID":"fc30260f-79d2-4307-ac41-0482974ad1e8","Type":"ContainerDied","Data":"abe3eb841370db13b923bad64284bcccef8b99ee7c3c26cee9693aec0e34c2e6"} Sep 09 11:00:44.181039 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:44.181017 2559 generic.go:356] "Generic (PLEG): container finished" podID="39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d" containerID="36bcc275cfa032b9507ec4d79f5daf100e1b86122072a7be3af9b82ef4d538ea" exitCode=0 Sep 09 11:00:44.181160 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:44.181080 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1eb21d-57fcdc7b5-n5fr8" event={"ID":"39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d","Type":"ContainerDied","Data":"36bcc275cfa032b9507ec4d79f5daf100e1b86122072a7be3af9b82ef4d538ea"} Sep 09 11:00:44.202980 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:44.202949 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1eb21d-57fcdc7b5-n5fr8" Sep 09 11:00:44.205669 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:44.205649 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1eb21d-846bcbb58-p64nc" Sep 09 11:00:44.333978 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:44.333937 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4bb16b3-8cef-4b5f-b215-7edcb11629d8" path="/var/lib/kubelet/pods/a4bb16b3-8cef-4b5f-b215-7edcb11629d8/volumes" Sep 09 11:00:44.353273 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:44.353239 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d-plugins-config-volume\") pod \"39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d\" (UID: \"39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d\") " Sep 09 11:00:44.353421 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:44.353301 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pb8rl\" (UniqueName: \"kubernetes.io/projected/39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d-kube-api-access-pb8rl\") pod \"39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d\" (UID: \"39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d\") " Sep 09 11:00:44.353421 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:44.353329 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc30260f-79d2-4307-ac41-0482974ad1e8-plugins-config-volume\") pod \"fc30260f-79d2-4307-ac41-0482974ad1e8\" (UID: \"fc30260f-79d2-4307-ac41-0482974ad1e8\") " Sep 09 11:00:44.353421 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:44.353396 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlqz2\" (UniqueName: \"kubernetes.io/projected/fc30260f-79d2-4307-ac41-0482974ad1e8-kube-api-access-xlqz2\") pod \"fc30260f-79d2-4307-ac41-0482974ad1e8\" (UID: \"fc30260f-79d2-4307-ac41-0482974ad1e8\") " Sep 09 11:00:44.353716 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:44.353689 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d" (UID: "39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 11:00:44.353851 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:44.353819 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc30260f-79d2-4307-ac41-0482974ad1e8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fc30260f-79d2-4307-ac41-0482974ad1e8" (UID: "fc30260f-79d2-4307-ac41-0482974ad1e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 11:00:44.355614 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:44.355588 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc30260f-79d2-4307-ac41-0482974ad1e8-kube-api-access-xlqz2" (OuterVolumeSpecName: "kube-api-access-xlqz2") pod "fc30260f-79d2-4307-ac41-0482974ad1e8" (UID: "fc30260f-79d2-4307-ac41-0482974ad1e8"). InnerVolumeSpecName "kube-api-access-xlqz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 11:00:44.355718 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:44.355678 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d-kube-api-access-pb8rl" (OuterVolumeSpecName: "kube-api-access-pb8rl") pod "39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d" (UID: "39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d"). InnerVolumeSpecName "kube-api-access-pb8rl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 11:00:44.454763 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:44.454665 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlqz2\" (UniqueName: \"kubernetes.io/projected/fc30260f-79d2-4307-ac41-0482974ad1e8-kube-api-access-xlqz2\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 11:00:44.454763 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:44.454700 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 11:00:44.454763 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:44.454716 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb8rl\" (UniqueName: \"kubernetes.io/projected/39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d-kube-api-access-pb8rl\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 11:00:44.454763 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:44.454730 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc30260f-79d2-4307-ac41-0482974ad1e8-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 11:00:45.187321 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:45.187286 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1eb21d-846bcbb58-p64nc" Sep 09 11:00:45.187822 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:45.187286 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1eb21d-846bcbb58-p64nc" event={"ID":"fc30260f-79d2-4307-ac41-0482974ad1e8","Type":"ContainerDied","Data":"8a00fcbabce016b0cd6578fb9f3d6f543a566ea1b152d32620b2c45e39c4c069"} Sep 09 11:00:45.187822 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:45.187449 2559 scope.go:117] "RemoveContainer" containerID="abe3eb841370db13b923bad64284bcccef8b99ee7c3c26cee9693aec0e34c2e6" Sep 09 11:00:45.188896 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:45.188630 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1eb21d-57fcdc7b5-n5fr8" event={"ID":"39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d","Type":"ContainerDied","Data":"0dc33ab488666563ab8c873630712a3a5e9e3c696cf646d083b7162bf9188dd7"} Sep 09 11:00:45.188896 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:45.188646 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1eb21d-57fcdc7b5-n5fr8" Sep 09 11:00:45.196822 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:45.196798 2559 scope.go:117] "RemoveContainer" containerID="36bcc275cfa032b9507ec4d79f5daf100e1b86122072a7be3af9b82ef4d538ea" Sep 09 11:00:45.216368 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:45.216334 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1eb21d-846bcbb58-p64nc"] Sep 09 11:00:45.220176 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:45.220151 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1eb21d-846bcbb58-p64nc"] Sep 09 11:00:45.231831 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:45.231805 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1eb21d-57fcdc7b5-n5fr8"] Sep 09 11:00:45.234433 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:45.234408 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1eb21d-57fcdc7b5-n5fr8"] Sep 09 11:00:46.334017 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:46.333979 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d" path="/var/lib/kubelet/pods/39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d/volumes" Sep 09 11:00:46.334536 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:46.334343 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc30260f-79d2-4307-ac41-0482974ad1e8" path="/var/lib/kubelet/pods/fc30260f-79d2-4307-ac41-0482974ad1e8/volumes" Sep 09 11:00:51.441900 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:51.441859 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-535b75-zqg6d"] Sep 09 11:00:51.442391 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:51.442206 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4bb16b3-8cef-4b5f-b215-7edcb11629d8" containerName="revoke-keys" Sep 09 11:00:51.442391 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:51.442218 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4bb16b3-8cef-4b5f-b215-7edcb11629d8" containerName="revoke-keys" Sep 09 11:00:51.442391 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:51.442237 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d" containerName="payload-processing" Sep 09 11:00:51.442391 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:51.442242 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d" containerName="payload-processing" Sep 09 11:00:51.442391 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:51.442251 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc30260f-79d2-4307-ac41-0482974ad1e8" containerName="payload-pre-processing" Sep 09 11:00:51.442391 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:51.442257 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc30260f-79d2-4307-ac41-0482974ad1e8" containerName="payload-pre-processing" Sep 09 11:00:51.442391 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:51.442313 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc30260f-79d2-4307-ac41-0482974ad1e8" containerName="payload-pre-processing" Sep 09 11:00:51.442391 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:51.442320 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="39e7e1a2-ebc5-4a25-9be5-e4da4d7e2a0d" containerName="payload-processing" Sep 09 11:00:51.442391 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:51.442327 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4bb16b3-8cef-4b5f-b215-7edcb11629d8" containerName="revoke-keys" Sep 09 11:00:51.447273 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:51.447242 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-535b75-zqg6d" Sep 09 11:00:51.449858 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:51.449830 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 11:00:51.450023 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:51.449835 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kq7qv\"" Sep 09 11:00:51.454214 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:51.454186 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-535b75-zqg6d"] Sep 09 11:00:51.614969 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:51.614926 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a3f8bf-3739-4156-87cc-5e9070824cf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-535b75-zqg6d\" (UID: \"22a3f8bf-3739-4156-87cc-5e9070824cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-535b75-zqg6d" Sep 09 11:00:51.716128 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:51.716020 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a3f8bf-3739-4156-87cc-5e9070824cf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-535b75-zqg6d\" (UID: \"22a3f8bf-3739-4156-87cc-5e9070824cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-535b75-zqg6d" Sep 09 11:00:51.716749 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:51.716724 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a3f8bf-3739-4156-87cc-5e9070824cf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-535b75-zqg6d\" (UID: \"22a3f8bf-3739-4156-87cc-5e9070824cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-535b75-zqg6d" Sep 09 11:00:51.759272 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:51.759223 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-535b75-zqg6d" Sep 09 11:00:52.032595 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:52.032567 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-535b75-zqg6d"] Sep 09 11:00:52.035224 ip-10-0-139-133 kubenswrapper[2559]: W0909 11:00:52.035199 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a3f8bf_3739_4156_87cc_5e9070824cf0.slice/crio-abf8d8da08f1080e56afb2a04574a818a9f7683f10199a9e333809fe6fa1aec6 WatchSource:0}: Error finding container abf8d8da08f1080e56afb2a04574a818a9f7683f10199a9e333809fe6fa1aec6: Status 404 returned error can't find the container with id abf8d8da08f1080e56afb2a04574a818a9f7683f10199a9e333809fe6fa1aec6 Sep 09 11:00:52.229140 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:52.229105 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-535b75-zqg6d" event={"ID":"22a3f8bf-3739-4156-87cc-5e9070824cf0","Type":"ContainerStarted","Data":"abf8d8da08f1080e56afb2a04574a818a9f7683f10199a9e333809fe6fa1aec6"} Sep 09 11:00:53.235707 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:53.235669 2559 generic.go:356] "Generic (PLEG): container finished" podID="22a3f8bf-3739-4156-87cc-5e9070824cf0" containerID="a7fb8602cd3f028e15bbb5d8e50170a54556f2fddf1cc67a2ac7a444149a0b55" exitCode=0 Sep 09 11:00:53.236182 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:53.235754 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-535b75-zqg6d" event={"ID":"22a3f8bf-3739-4156-87cc-5e9070824cf0","Type":"ContainerDied","Data":"a7fb8602cd3f028e15bbb5d8e50170a54556f2fddf1cc67a2ac7a444149a0b55"} Sep 09 11:00:54.383027 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:54.383001 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-535b75-zqg6d" Sep 09 11:00:54.542154 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:54.542049 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a3f8bf-3739-4156-87cc-5e9070824cf0-maas-api-service-ca\") pod \"22a3f8bf-3739-4156-87cc-5e9070824cf0\" (UID: \"22a3f8bf-3739-4156-87cc-5e9070824cf0\") " Sep 09 11:00:54.542431 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:54.542406 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a3f8bf-3739-4156-87cc-5e9070824cf0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "22a3f8bf-3739-4156-87cc-5e9070824cf0" (UID: "22a3f8bf-3739-4156-87cc-5e9070824cf0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 11:00:54.643686 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:54.643647 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a3f8bf-3739-4156-87cc-5e9070824cf0-maas-api-service-ca\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 11:00:55.248832 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:55.248785 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-535b75-zqg6d" event={"ID":"22a3f8bf-3739-4156-87cc-5e9070824cf0","Type":"ContainerDied","Data":"abf8d8da08f1080e56afb2a04574a818a9f7683f10199a9e333809fe6fa1aec6"} Sep 09 11:00:55.248832 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:55.248826 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-535b75-zqg6d" Sep 09 11:00:55.248832 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:00:55.248839 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abf8d8da08f1080e56afb2a04574a818a9f7683f10199a9e333809fe6fa1aec6" Sep 09 11:01:00.411278 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:00.411245 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d22ff_0a56b285-3f3c-4f6d-a939-8a95ae00e8da/ovn-acl-logging/0.log" Sep 09 11:01:00.417633 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:00.417609 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d22ff_0a56b285-3f3c-4f6d-a939-8a95ae00e8da/ovn-acl-logging/0.log" Sep 09 11:01:01.996424 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:01.996364 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-535b75-29815860-v2267"] Sep 09 11:01:01.999892 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:01.999865 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-535b75-29815860-v2267"] Sep 09 11:01:02.017443 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:02.017408 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-535b75-5df86cb7f9-npkvs"] Sep 09 11:01:02.017726 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:02.017681 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-535b75-5df86cb7f9-npkvs" podUID="5460687d-4874-49df-8f22-38e79231b6a4" containerName="payload-processing" containerID="cri-o://bfbd32ed5484e5dcd89253bcda4e15a7ae21d2cd0ee00a698d1ba22426c75593" gracePeriod=30 Sep 09 11:01:02.027885 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:02.027853 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-535b75-78f95fcfb6-5lpzd"] Sep 09 11:01:02.028224 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:02.028109 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-535b75-78f95fcfb6-5lpzd" podUID="045ab401-6d45-4953-a1f0-7e590bf20c24" containerName="payload-pre-processing" containerID="cri-o://90f32e03a31c89046a50545d8c025a994d1bc3f368e67b455a5c1162378c5681" gracePeriod=30 Sep 09 11:01:02.231373 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:02.231337 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-535b75-zqg6d"] Sep 09 11:01:02.233169 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:02.233131 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-535b75-zqg6d"] Sep 09 11:01:02.287482 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:02.287309 2559 generic.go:356] "Generic (PLEG): container finished" podID="045ab401-6d45-4953-a1f0-7e590bf20c24" containerID="90f32e03a31c89046a50545d8c025a994d1bc3f368e67b455a5c1162378c5681" exitCode=0 Sep 09 11:01:02.287482 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:02.287376 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-535b75-78f95fcfb6-5lpzd" event={"ID":"045ab401-6d45-4953-a1f0-7e590bf20c24","Type":"ContainerDied","Data":"90f32e03a31c89046a50545d8c025a994d1bc3f368e67b455a5c1162378c5681"} Sep 09 11:01:02.289005 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:02.288978 2559 generic.go:356] "Generic (PLEG): container finished" podID="5460687d-4874-49df-8f22-38e79231b6a4" containerID="bfbd32ed5484e5dcd89253bcda4e15a7ae21d2cd0ee00a698d1ba22426c75593" exitCode=0 Sep 09 11:01:02.289111 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:02.289044 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-535b75-5df86cb7f9-npkvs" event={"ID":"5460687d-4874-49df-8f22-38e79231b6a4","Type":"ContainerDied","Data":"bfbd32ed5484e5dcd89253bcda4e15a7ae21d2cd0ee00a698d1ba22426c75593"} Sep 09 11:01:02.310692 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:02.310661 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-535b75-5df86cb7f9-npkvs" Sep 09 11:01:02.313848 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:02.313828 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-535b75-78f95fcfb6-5lpzd" Sep 09 11:01:02.335156 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:02.334965 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="000569a5-038c-421d-b725-35765e669175" path="/var/lib/kubelet/pods/000569a5-038c-421d-b725-35765e669175/volumes" Sep 09 11:01:02.335398 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:02.335376 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22a3f8bf-3739-4156-87cc-5e9070824cf0" path="/var/lib/kubelet/pods/22a3f8bf-3739-4156-87cc-5e9070824cf0/volumes" Sep 09 11:01:02.507079 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:02.507040 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5460687d-4874-49df-8f22-38e79231b6a4-plugins-config-volume\") pod \"5460687d-4874-49df-8f22-38e79231b6a4\" (UID: \"5460687d-4874-49df-8f22-38e79231b6a4\") " Sep 09 11:01:02.507079 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:02.507084 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6vds\" (UniqueName: \"kubernetes.io/projected/045ab401-6d45-4953-a1f0-7e590bf20c24-kube-api-access-h6vds\") pod \"045ab401-6d45-4953-a1f0-7e590bf20c24\" (UID: \"045ab401-6d45-4953-a1f0-7e590bf20c24\") " Sep 09 11:01:02.507360 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:02.507108 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/045ab401-6d45-4953-a1f0-7e590bf20c24-plugins-config-volume\") pod \"045ab401-6d45-4953-a1f0-7e590bf20c24\" (UID: \"045ab401-6d45-4953-a1f0-7e590bf20c24\") " Sep 09 11:01:02.507360 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:02.507125 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wp52h\" (UniqueName: \"kubernetes.io/projected/5460687d-4874-49df-8f22-38e79231b6a4-kube-api-access-wp52h\") pod \"5460687d-4874-49df-8f22-38e79231b6a4\" (UID: \"5460687d-4874-49df-8f22-38e79231b6a4\") " Sep 09 11:01:02.507588 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:02.507554 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/045ab401-6d45-4953-a1f0-7e590bf20c24-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "045ab401-6d45-4953-a1f0-7e590bf20c24" (UID: "045ab401-6d45-4953-a1f0-7e590bf20c24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 11:01:02.507727 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:02.507555 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5460687d-4874-49df-8f22-38e79231b6a4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5460687d-4874-49df-8f22-38e79231b6a4" (UID: "5460687d-4874-49df-8f22-38e79231b6a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 11:01:02.509301 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:02.509281 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/045ab401-6d45-4953-a1f0-7e590bf20c24-kube-api-access-h6vds" (OuterVolumeSpecName: "kube-api-access-h6vds") pod "045ab401-6d45-4953-a1f0-7e590bf20c24" (UID: "045ab401-6d45-4953-a1f0-7e590bf20c24"). InnerVolumeSpecName "kube-api-access-h6vds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 11:01:02.509369 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:02.509328 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5460687d-4874-49df-8f22-38e79231b6a4-kube-api-access-wp52h" (OuterVolumeSpecName: "kube-api-access-wp52h") pod "5460687d-4874-49df-8f22-38e79231b6a4" (UID: "5460687d-4874-49df-8f22-38e79231b6a4"). InnerVolumeSpecName "kube-api-access-wp52h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 11:01:02.608479 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:02.608433 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5460687d-4874-49df-8f22-38e79231b6a4-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 11:01:02.608479 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:02.608468 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6vds\" (UniqueName: \"kubernetes.io/projected/045ab401-6d45-4953-a1f0-7e590bf20c24-kube-api-access-h6vds\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 11:01:02.608479 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:02.608478 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/045ab401-6d45-4953-a1f0-7e590bf20c24-plugins-config-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 11:01:02.608479 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:02.608489 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp52h\" (UniqueName: \"kubernetes.io/projected/5460687d-4874-49df-8f22-38e79231b6a4-kube-api-access-wp52h\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 11:01:03.295824 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:03.295739 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-535b75-78f95fcfb6-5lpzd" Sep 09 11:01:03.296243 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:03.295728 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-535b75-78f95fcfb6-5lpzd" event={"ID":"045ab401-6d45-4953-a1f0-7e590bf20c24","Type":"ContainerDied","Data":"0f931d6f4a5534f504b4695c15275bf12ce3375c5d273cd311282612fd165c90"} Sep 09 11:01:03.296243 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:03.295923 2559 scope.go:117] "RemoveContainer" containerID="90f32e03a31c89046a50545d8c025a994d1bc3f368e67b455a5c1162378c5681" Sep 09 11:01:03.297367 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:03.297302 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-535b75-5df86cb7f9-npkvs" Sep 09 11:01:03.297367 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:03.297329 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-535b75-5df86cb7f9-npkvs" event={"ID":"5460687d-4874-49df-8f22-38e79231b6a4","Type":"ContainerDied","Data":"b839e761d4dafe9e5fd84eebc61abc793913f6b295e90e4672f56654f87b6c1c"} Sep 09 11:01:03.307039 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:03.307019 2559 scope.go:117] "RemoveContainer" containerID="bfbd32ed5484e5dcd89253bcda4e15a7ae21d2cd0ee00a698d1ba22426c75593" Sep 09 11:01:03.319470 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:03.319439 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-535b75-78f95fcfb6-5lpzd"] Sep 09 11:01:03.331054 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:03.331025 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-535b75-78f95fcfb6-5lpzd"] Sep 09 11:01:03.345545 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:03.345488 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-535b75-5df86cb7f9-npkvs"] Sep 09 11:01:03.348590 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:03.348564 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-535b75-5df86cb7f9-npkvs"] Sep 09 11:01:04.333831 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:04.333798 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="045ab401-6d45-4953-a1f0-7e590bf20c24" path="/var/lib/kubelet/pods/045ab401-6d45-4953-a1f0-7e590bf20c24/volumes" Sep 09 11:01:04.334258 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:01:04.334243 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5460687d-4874-49df-8f22-38e79231b6a4" path="/var/lib/kubelet/pods/5460687d-4874-49df-8f22-38e79231b6a4/volumes" Sep 09 11:02:00.722046 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:02:00.722011 2559 scope.go:117] "RemoveContainer" containerID="5a06433ea37fbe297f37b9978c1fc1ac514e95cf57cf41562cd21753b5dc5329" Sep 09 11:03:00.750669 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:03:00.750635 2559 scope.go:117] "RemoveContainer" containerID="60069030d7540c7fcc8930d9661fb24512fba583ed0c687cca2e49dd6290f601" Sep 09 11:03:00.759541 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:03:00.759515 2559 scope.go:117] "RemoveContainer" containerID="eb00429de281b049701c9a2631c6e5a4da5e2624077350872b7360cc9decaffe" Sep 09 11:03:00.767569 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:03:00.767547 2559 scope.go:117] "RemoveContainer" containerID="7f758de8ba9335a5b7d7f1a7a5d28b7c4604cd77cea002fb9dedf1608b422e32" Sep 09 11:03:46.874665 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:03:46.874586 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hk5f5"] Sep 09 11:03:46.875224 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:03:46.874842 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hk5f5" podUID="9ae40ea5-d767-4a0d-a27b-5a246d66a83c" containerName="manager" containerID="cri-o://856b9208c2571b532035938dc396c62bb56dcf3a55e7c351b0c33bee815ac50f" gracePeriod=10 Sep 09 11:03:47.719169 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:03:47.719142 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hk5f5" Sep 09 11:03:47.809908 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:03:47.809812 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2t74p\" (UniqueName: \"kubernetes.io/projected/9ae40ea5-d767-4a0d-a27b-5a246d66a83c-kube-api-access-2t74p\") pod \"9ae40ea5-d767-4a0d-a27b-5a246d66a83c\" (UID: \"9ae40ea5-d767-4a0d-a27b-5a246d66a83c\") " Sep 09 11:03:47.809908 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:03:47.809859 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ae40ea5-d767-4a0d-a27b-5a246d66a83c-extensions-socket-volume\") pod \"9ae40ea5-d767-4a0d-a27b-5a246d66a83c\" (UID: \"9ae40ea5-d767-4a0d-a27b-5a246d66a83c\") " Sep 09 11:03:47.810244 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:03:47.810217 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae40ea5-d767-4a0d-a27b-5a246d66a83c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9ae40ea5-d767-4a0d-a27b-5a246d66a83c" (UID: "9ae40ea5-d767-4a0d-a27b-5a246d66a83c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 11:03:47.812016 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:03:47.811989 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae40ea5-d767-4a0d-a27b-5a246d66a83c-kube-api-access-2t74p" (OuterVolumeSpecName: "kube-api-access-2t74p") pod "9ae40ea5-d767-4a0d-a27b-5a246d66a83c" (UID: "9ae40ea5-d767-4a0d-a27b-5a246d66a83c"). InnerVolumeSpecName "kube-api-access-2t74p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 11:03:47.911377 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:03:47.911332 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t74p\" (UniqueName: \"kubernetes.io/projected/9ae40ea5-d767-4a0d-a27b-5a246d66a83c-kube-api-access-2t74p\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 11:03:47.911377 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:03:47.911369 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ae40ea5-d767-4a0d-a27b-5a246d66a83c-extensions-socket-volume\") on node \"ip-10-0-139-133.ec2.internal\" DevicePath \"\"" Sep 09 11:03:48.136239 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:03:48.136203 2559 generic.go:356] "Generic (PLEG): container finished" podID="9ae40ea5-d767-4a0d-a27b-5a246d66a83c" containerID="856b9208c2571b532035938dc396c62bb56dcf3a55e7c351b0c33bee815ac50f" exitCode=0 Sep 09 11:03:48.136432 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:03:48.136251 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hk5f5" Sep 09 11:03:48.136432 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:03:48.136286 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hk5f5" event={"ID":"9ae40ea5-d767-4a0d-a27b-5a246d66a83c","Type":"ContainerDied","Data":"856b9208c2571b532035938dc396c62bb56dcf3a55e7c351b0c33bee815ac50f"} Sep 09 11:03:48.136432 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:03:48.136339 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hk5f5" event={"ID":"9ae40ea5-d767-4a0d-a27b-5a246d66a83c","Type":"ContainerDied","Data":"b867ae70df78f97a3432f35e53a2423a3b7833306a5e140fcefd0bf446730dbc"} Sep 09 11:03:48.136432 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:03:48.136362 2559 scope.go:117] "RemoveContainer" containerID="856b9208c2571b532035938dc396c62bb56dcf3a55e7c351b0c33bee815ac50f" Sep 09 11:03:48.145593 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:03:48.145567 2559 scope.go:117] "RemoveContainer" containerID="856b9208c2571b532035938dc396c62bb56dcf3a55e7c351b0c33bee815ac50f" Sep 09 11:03:48.145914 ip-10-0-139-133 kubenswrapper[2559]: E0909 11:03:48.145895 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"856b9208c2571b532035938dc396c62bb56dcf3a55e7c351b0c33bee815ac50f\": container with ID starting with 856b9208c2571b532035938dc396c62bb56dcf3a55e7c351b0c33bee815ac50f not found: ID does not exist" containerID="856b9208c2571b532035938dc396c62bb56dcf3a55e7c351b0c33bee815ac50f" Sep 09 11:03:48.145962 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:03:48.145924 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"856b9208c2571b532035938dc396c62bb56dcf3a55e7c351b0c33bee815ac50f"} err="failed to get container status \"856b9208c2571b532035938dc396c62bb56dcf3a55e7c351b0c33bee815ac50f\": rpc error: code = NotFound desc = could not find container \"856b9208c2571b532035938dc396c62bb56dcf3a55e7c351b0c33bee815ac50f\": container with ID starting with 856b9208c2571b532035938dc396c62bb56dcf3a55e7c351b0c33bee815ac50f not found: ID does not exist" Sep 09 11:03:48.160466 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:03:48.160431 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hk5f5"] Sep 09 11:03:48.165043 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:03:48.165015 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hk5f5"] Sep 09 11:03:48.334654 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:03:48.334621 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ae40ea5-d767-4a0d-a27b-5a246d66a83c" path="/var/lib/kubelet/pods/9ae40ea5-d767-4a0d-a27b-5a246d66a83c/volumes" Sep 09 11:04:47.576407 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:04:47.576371 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p2psb"] Sep 09 11:04:47.577044 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:04:47.576863 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22a3f8bf-3739-4156-87cc-5e9070824cf0" containerName="revoke-keys" Sep 09 11:04:47.577044 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:04:47.576882 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="22a3f8bf-3739-4156-87cc-5e9070824cf0" containerName="revoke-keys" Sep 09 11:04:47.577044 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:04:47.576900 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="045ab401-6d45-4953-a1f0-7e590bf20c24" containerName="payload-pre-processing" Sep 09 11:04:47.577044 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:04:47.576910 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="045ab401-6d45-4953-a1f0-7e590bf20c24" containerName="payload-pre-processing" Sep 09 11:04:47.577044 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:04:47.576928 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ae40ea5-d767-4a0d-a27b-5a246d66a83c" containerName="manager" Sep 09 11:04:47.577044 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:04:47.576936 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ae40ea5-d767-4a0d-a27b-5a246d66a83c" containerName="manager" Sep 09 11:04:47.577044 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:04:47.576961 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5460687d-4874-49df-8f22-38e79231b6a4" containerName="payload-processing" Sep 09 11:04:47.577044 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:04:47.576969 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5460687d-4874-49df-8f22-38e79231b6a4" containerName="payload-processing" Sep 09 11:04:47.577460 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:04:47.577049 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5460687d-4874-49df-8f22-38e79231b6a4" containerName="payload-processing" Sep 09 11:04:47.577460 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:04:47.577064 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ae40ea5-d767-4a0d-a27b-5a246d66a83c" containerName="manager" Sep 09 11:04:47.577460 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:04:47.577071 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="22a3f8bf-3739-4156-87cc-5e9070824cf0" containerName="revoke-keys" Sep 09 11:04:47.577460 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:04:47.577081 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="045ab401-6d45-4953-a1f0-7e590bf20c24" containerName="payload-pre-processing" Sep 09 11:04:47.580039 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:04:47.580019 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p2psb" Sep 09 11:04:47.583281 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:04:47.583261 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-x9wq2\"" Sep 09 11:04:47.608415 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:04:47.608382 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p2psb"] Sep 09 11:04:47.704878 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:04:47.704833 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f7550d6-1d3f-403a-a6f5-0a6fb7f58d85-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-p2psb\" (UID: \"6f7550d6-1d3f-403a-a6f5-0a6fb7f58d85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p2psb" Sep 09 11:04:47.704878 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:04:47.704879 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9brkd\" (UniqueName: \"kubernetes.io/projected/6f7550d6-1d3f-403a-a6f5-0a6fb7f58d85-kube-api-access-9brkd\") pod \"kuadrant-operator-controller-manager-5769f99d9f-p2psb\" (UID: \"6f7550d6-1d3f-403a-a6f5-0a6fb7f58d85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p2psb" Sep 09 11:04:47.805442 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:04:47.805392 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f7550d6-1d3f-403a-a6f5-0a6fb7f58d85-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-p2psb\" (UID: \"6f7550d6-1d3f-403a-a6f5-0a6fb7f58d85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p2psb" Sep 09 11:04:47.805442 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:04:47.805437 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9brkd\" (UniqueName: \"kubernetes.io/projected/6f7550d6-1d3f-403a-a6f5-0a6fb7f58d85-kube-api-access-9brkd\") pod \"kuadrant-operator-controller-manager-5769f99d9f-p2psb\" (UID: \"6f7550d6-1d3f-403a-a6f5-0a6fb7f58d85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p2psb" Sep 09 11:04:47.805824 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:04:47.805797 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f7550d6-1d3f-403a-a6f5-0a6fb7f58d85-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-p2psb\" (UID: \"6f7550d6-1d3f-403a-a6f5-0a6fb7f58d85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p2psb" Sep 09 11:04:47.817882 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:04:47.817849 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9brkd\" (UniqueName: \"kubernetes.io/projected/6f7550d6-1d3f-403a-a6f5-0a6fb7f58d85-kube-api-access-9brkd\") pod \"kuadrant-operator-controller-manager-5769f99d9f-p2psb\" (UID: \"6f7550d6-1d3f-403a-a6f5-0a6fb7f58d85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p2psb" Sep 09 11:04:47.890489 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:04:47.890388 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p2psb" Sep 09 11:04:48.181254 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:04:48.181144 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p2psb"] Sep 09 11:04:48.184386 ip-10-0-139-133 kubenswrapper[2559]: W0909 11:04:48.184348 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f7550d6_1d3f_403a_a6f5_0a6fb7f58d85.slice/crio-5a4b890468067390c0cefd1853441364720c31526c707a93c021e6921d002503 WatchSource:0}: Error finding container 5a4b890468067390c0cefd1853441364720c31526c707a93c021e6921d002503: Status 404 returned error can't find the container with id 5a4b890468067390c0cefd1853441364720c31526c707a93c021e6921d002503 Sep 09 11:04:48.430828 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:04:48.430790 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p2psb" event={"ID":"6f7550d6-1d3f-403a-a6f5-0a6fb7f58d85","Type":"ContainerStarted","Data":"b9e18de6d1ab01f29c3ebc9d1b66606b0adc1596ac16d81e6b1740669745e270"} Sep 09 11:04:48.431006 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:04:48.430853 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p2psb" event={"ID":"6f7550d6-1d3f-403a-a6f5-0a6fb7f58d85","Type":"ContainerStarted","Data":"5a4b890468067390c0cefd1853441364720c31526c707a93c021e6921d002503"} Sep 09 11:04:48.431006 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:04:48.430909 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p2psb" Sep 09 11:04:48.457655 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:04:48.457558 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p2psb" podStartSLOduration=1.457540829 podStartE2EDuration="1.457540829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 11:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 11:04:48.45664378 +0000 UTC m=+2328.631246116" watchObservedRunningTime="2026-09-09 11:04:48.457540829 +0000 UTC m=+2328.632143153" Sep 09 11:04:59.438672 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:04:59.438584 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p2psb" Sep 09 11:05:00.804278 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:05:00.804242 2559 scope.go:117] "RemoveContainer" containerID="1ebbd3aac841af9866d03c0aa094422a311938398eecedd5c68fb26c9b6b3f78" Sep 09 11:06:00.440517 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:00.440476 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d22ff_0a56b285-3f3c-4f6d-a939-8a95ae00e8da/ovn-acl-logging/0.log" Sep 09 11:06:00.449732 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:00.449709 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d22ff_0a56b285-3f3c-4f6d-a939-8a95ae00e8da/ovn-acl-logging/0.log" Sep 09 11:06:00.824106 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:00.824017 2559 scope.go:117] "RemoveContainer" containerID="ca2d67ca3ebbcc7611508ddb540adcbff88f4dbb4cf269575bc993660c9440c6" Sep 09 11:06:00.832565 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:00.832541 2559 scope.go:117] "RemoveContainer" containerID="3fb6b2fcf39ff2b988018f6858283f3fa4b04b4c794741a7149d42b008de74f6" Sep 09 11:06:00.840390 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:00.840367 2559 scope.go:117] "RemoveContainer" containerID="56e4cd032396a0a6aadc1993310f84efa5be27456f47f2181764c8bdb4ddcd73" Sep 09 11:06:21.225993 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:21.225955 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-ff785c76b-lzp24"] Sep 09 11:06:21.229297 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:21.229279 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-ff785c76b-lzp24" Sep 09 11:06:21.232558 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:21.232537 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-k8bm4\"" Sep 09 11:06:21.233976 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:21.233952 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 09 11:06:21.234093 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:21.233990 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 11:06:21.234093 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:21.234010 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 09 11:06:21.234093 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:21.234069 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 09 11:06:21.264809 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:21.264764 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-ff785c76b-lzp24"] Sep 09 11:06:21.281145 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:21.281114 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f655eaac-be22-4e9d-ac69-da861a879c13-openshift-service-ca\") pod \"maas-api-ff785c76b-lzp24\" (UID: \"f655eaac-be22-4e9d-ac69-da861a879c13\") " pod="odh-ai-gateway-infra/maas-api-ff785c76b-lzp24" Sep 09 11:06:21.281145 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:21.281147 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f655eaac-be22-4e9d-ac69-da861a879c13-metrics-certs\") pod \"maas-api-ff785c76b-lzp24\" (UID: \"f655eaac-be22-4e9d-ac69-da861a879c13\") " pod="odh-ai-gateway-infra/maas-api-ff785c76b-lzp24" Sep 09 11:06:21.281368 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:21.281188 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f655eaac-be22-4e9d-ac69-da861a879c13-maas-api-tls\") pod \"maas-api-ff785c76b-lzp24\" (UID: \"f655eaac-be22-4e9d-ac69-da861a879c13\") " pod="odh-ai-gateway-infra/maas-api-ff785c76b-lzp24" Sep 09 11:06:21.281368 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:21.281281 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj8zl\" (UniqueName: \"kubernetes.io/projected/f655eaac-be22-4e9d-ac69-da861a879c13-kube-api-access-mj8zl\") pod \"maas-api-ff785c76b-lzp24\" (UID: \"f655eaac-be22-4e9d-ac69-da861a879c13\") " pod="odh-ai-gateway-infra/maas-api-ff785c76b-lzp24" Sep 09 11:06:21.382063 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:21.382024 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f655eaac-be22-4e9d-ac69-da861a879c13-openshift-service-ca\") pod \"maas-api-ff785c76b-lzp24\" (UID: \"f655eaac-be22-4e9d-ac69-da861a879c13\") " pod="odh-ai-gateway-infra/maas-api-ff785c76b-lzp24" Sep 09 11:06:21.382286 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:21.382074 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f655eaac-be22-4e9d-ac69-da861a879c13-metrics-certs\") pod \"maas-api-ff785c76b-lzp24\" (UID: \"f655eaac-be22-4e9d-ac69-da861a879c13\") " pod="odh-ai-gateway-infra/maas-api-ff785c76b-lzp24" Sep 09 11:06:21.382286 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:21.382263 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f655eaac-be22-4e9d-ac69-da861a879c13-maas-api-tls\") pod \"maas-api-ff785c76b-lzp24\" (UID: \"f655eaac-be22-4e9d-ac69-da861a879c13\") " pod="odh-ai-gateway-infra/maas-api-ff785c76b-lzp24" Sep 09 11:06:21.382445 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:21.382312 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj8zl\" (UniqueName: \"kubernetes.io/projected/f655eaac-be22-4e9d-ac69-da861a879c13-kube-api-access-mj8zl\") pod \"maas-api-ff785c76b-lzp24\" (UID: \"f655eaac-be22-4e9d-ac69-da861a879c13\") " pod="odh-ai-gateway-infra/maas-api-ff785c76b-lzp24" Sep 09 11:06:21.382840 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:21.382817 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f655eaac-be22-4e9d-ac69-da861a879c13-openshift-service-ca\") pod \"maas-api-ff785c76b-lzp24\" (UID: \"f655eaac-be22-4e9d-ac69-da861a879c13\") " pod="odh-ai-gateway-infra/maas-api-ff785c76b-lzp24" Sep 09 11:06:21.384681 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:21.384660 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f655eaac-be22-4e9d-ac69-da861a879c13-metrics-certs\") pod \"maas-api-ff785c76b-lzp24\" (UID: \"f655eaac-be22-4e9d-ac69-da861a879c13\") " pod="odh-ai-gateway-infra/maas-api-ff785c76b-lzp24" Sep 09 11:06:21.384808 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:21.384667 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f655eaac-be22-4e9d-ac69-da861a879c13-maas-api-tls\") pod \"maas-api-ff785c76b-lzp24\" (UID: \"f655eaac-be22-4e9d-ac69-da861a879c13\") " pod="odh-ai-gateway-infra/maas-api-ff785c76b-lzp24" Sep 09 11:06:21.393941 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:21.393914 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj8zl\" (UniqueName: \"kubernetes.io/projected/f655eaac-be22-4e9d-ac69-da861a879c13-kube-api-access-mj8zl\") pod \"maas-api-ff785c76b-lzp24\" (UID: \"f655eaac-be22-4e9d-ac69-da861a879c13\") " pod="odh-ai-gateway-infra/maas-api-ff785c76b-lzp24" Sep 09 11:06:21.540159 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:21.540061 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-ff785c76b-lzp24" Sep 09 11:06:21.819370 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:21.819341 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-ff785c76b-lzp24"] Sep 09 11:06:21.822125 ip-10-0-139-133 kubenswrapper[2559]: W0909 11:06:21.822096 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf655eaac_be22_4e9d_ac69_da861a879c13.slice/crio-b29e585d9e5620868b5f3a44936f6bf607297819fb230feab962c5bda4442dfb WatchSource:0}: Error finding container b29e585d9e5620868b5f3a44936f6bf607297819fb230feab962c5bda4442dfb: Status 404 returned error can't find the container with id b29e585d9e5620868b5f3a44936f6bf607297819fb230feab962c5bda4442dfb Sep 09 11:06:21.823384 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:21.823362 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 11:06:21.896562 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:21.896519 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-ff785c76b-lzp24" event={"ID":"f655eaac-be22-4e9d-ac69-da861a879c13","Type":"ContainerStarted","Data":"b29e585d9e5620868b5f3a44936f6bf607297819fb230feab962c5bda4442dfb"} Sep 09 11:06:27.931165 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:27.931120 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-ff785c76b-lzp24" event={"ID":"f655eaac-be22-4e9d-ac69-da861a879c13","Type":"ContainerStarted","Data":"5e8017fa093bfddf583e35deb9d53147c8ac9cff1f8e3658d7f7b664b15eb29f"} Sep 09 11:06:27.931647 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:27.931256 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-ff785c76b-lzp24" Sep 09 11:06:27.956374 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:27.956320 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-ff785c76b-lzp24" podStartSLOduration=1.313009201 podStartE2EDuration="6.956305117s" totalImagesPullingTime="5.643295916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 11:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 11:06:27.954784037 +0000 UTC m=+2428.129386362" watchObservedRunningTime="2026-09-09 11:06:27.956305117 +0000 UTC m=+2428.130907442" Sep 09 11:06:33.940821 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:33.940788 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-ff785c76b-lzp24" Sep 09 11:06:53.356460 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:53.356425 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-k45xt_2aa16cb0-c44e-48cd-986e-cc46caef5437/manager/0.log" Sep 09 11:06:53.494836 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:53.494803 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-xd7gg_3d352779-62ce-4e83-ac40-5de8a05026cc/manager/0.log" Sep 09 11:06:54.152629 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:54.152596 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-544cbf49fb-2nr65_a2658a6e-38ca-405b-bb5c-5767972f0fc9/copy-manifests/0.log" Sep 09 11:06:54.172780 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:54.172751 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-544cbf49fb-2nr65_a2658a6e-38ca-405b-bb5c-5767972f0fc9/manager/0.log" Sep 09 11:06:54.804739 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:54.804712 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-ff785c76b-lzp24_f655eaac-be22-4e9d-ac69-da861a879c13/maas-api/0.log" Sep 09 11:06:54.924386 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:54.924357 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29815845-276hk_19a594f7-1a3f-4e60-b0ea-69ceb6221560/cleanup/0.log" Sep 09 11:06:55.043381 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:55.043353 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29815860-k5dlj_55468590-26e0-40b1-92db-dd92657ebd47/cleanup/0.log" Sep 09 11:06:56.464248 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:56.464217 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-p2psb_6f7550d6-1d3f-403a-a6f5-0a6fb7f58d85/manager/0.log" Sep 09 11:06:56.781521 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:56.781426 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-g2tnk_2e5f3a17-b2c5-4e9e-bca2-00c821c9b0a6/manager/0.log" Sep 09 11:06:57.260604 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:57.260571 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-v2t7d_af4f8698-6435-4c45-8a43-0d2eba16ce43/discovery/0.log" Sep 09 11:06:57.473644 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:57.473610 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75cd99c5fc-fqj8r_c719a5cf-09c3-4950-916a-33922d99dad4/kube-auth-proxy/0.log" Sep 09 11:06:57.813930 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:57.813899 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-x4njj_1cdfe780-23e6-4fa2-bb10-dd57f4c5c25b/payload-processing/0.log" Sep 09 11:06:58.408933 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:58.408902 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-k26kt_b38da3c8-0e6e-4ffc-b5ec-0d8d8f1ab1db/main/0.log" Sep 09 11:06:58.632613 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:06:58.632581 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-zpsd9_2d9be21b-3c67-4186-903f-8d6a7e76038c/main/0.log" Sep 09 11:07:00.870166 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:00.870135 2559 scope.go:117] "RemoveContainer" containerID="a7fb8602cd3f028e15bbb5d8e50170a54556f2fddf1cc67a2ac7a444149a0b55" Sep 09 11:07:00.878190 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:00.878167 2559 scope.go:117] "RemoveContainer" containerID="e2ed2c48ea68e545a32eab95aab587b8b676ba13ea681a68468f6f95ad3043ad" Sep 09 11:07:00.885421 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:00.885401 2559 scope.go:117] "RemoveContainer" containerID="72c4ab9c71f291ac7588542979102fc2e546a7ac6b262ac27610664959256170" Sep 09 11:07:08.077809 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:08.077775 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-k9fqd_b697c056-99a2-4c91-a7c3-1f787553973e/global-pull-secret-syncer/0.log" Sep 09 11:07:08.168971 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:08.168935 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-l9ttv_3ceff104-498a-49b9-a2b1-0d9de58ebca3/konnectivity-agent/0.log" Sep 09 11:07:08.304515 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:08.304485 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-133.ec2.internal_cecf6a195bd7af1b2519a96aa95ed769/haproxy/0.log" Sep 09 11:07:12.967613 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:12.967576 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-p2psb_6f7550d6-1d3f-403a-a6f5-0a6fb7f58d85/manager/0.log" Sep 09 11:07:13.096606 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:13.096559 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-g2tnk_2e5f3a17-b2c5-4e9e-bca2-00c821c9b0a6/manager/0.log" Sep 09 11:07:15.471757 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:15.471718 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-qnjb6_3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3/kube-state-metrics/0.log" Sep 09 11:07:15.521608 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:15.521579 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-qnjb6_3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3/kube-rbac-proxy-main/0.log" Sep 09 11:07:15.566992 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:15.566962 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-qnjb6_3a0e3c69-32c0-4328-8b0e-41c17c0fe6b3/kube-rbac-proxy-self/0.log" Sep 09 11:07:15.775983 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:15.775908 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8jhbg_e0d829bc-3ccb-4fd7-ab82-bab2b9128c88/node-exporter/0.log" Sep 09 11:07:15.831316 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:15.831287 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8jhbg_e0d829bc-3ccb-4fd7-ab82-bab2b9128c88/kube-rbac-proxy/0.log" Sep 09 11:07:15.887167 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:15.887137 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8jhbg_e0d829bc-3ccb-4fd7-ab82-bab2b9128c88/init-textfile/0.log" Sep 09 11:07:16.354288 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:16.354263 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-4p88h_2a60cc00-e367-4273-8317-e3e33e871abc/kube-rbac-proxy-main/0.log" Sep 09 11:07:16.403242 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:16.403209 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-4p88h_2a60cc00-e367-4273-8317-e3e33e871abc/kube-rbac-proxy-self/0.log" Sep 09 11:07:16.442934 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:16.442902 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-4p88h_2a60cc00-e367-4273-8317-e3e33e871abc/openshift-state-metrics/0.log" Sep 09 11:07:16.885679 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:16.885647 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-8fs4c_6401dd74-94ae-484b-bf82-fcd8d4ff6898/prometheus-operator-admission-webhook/0.log" Sep 09 11:07:17.448835 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:17.448800 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-86rrh/perf-node-gather-daemonset-jrjst"] Sep 09 11:07:17.452324 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:17.452303 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-jrjst" Sep 09 11:07:17.456633 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:17.456603 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-86rrh\"/\"kube-root-ca.crt\"" Sep 09 11:07:17.457454 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:17.457433 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-86rrh\"/\"default-dockercfg-l6fff\"" Sep 09 11:07:17.457521 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:17.457434 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-86rrh\"/\"openshift-service-ca.crt\"" Sep 09 11:07:17.464018 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:17.463989 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-86rrh/perf-node-gather-daemonset-jrjst"] Sep 09 11:07:17.574776 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:17.574736 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5d0c79f4-962d-4736-9953-2596e22ada2f-proc\") pod \"perf-node-gather-daemonset-jrjst\" (UID: \"5d0c79f4-962d-4736-9953-2596e22ada2f\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-jrjst" Sep 09 11:07:17.574776 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:17.574775 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5d0c79f4-962d-4736-9953-2596e22ada2f-podres\") pod \"perf-node-gather-daemonset-jrjst\" (UID: \"5d0c79f4-962d-4736-9953-2596e22ada2f\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-jrjst" Sep 09 11:07:17.575043 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:17.574863 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d0c79f4-962d-4736-9953-2596e22ada2f-sys\") pod \"perf-node-gather-daemonset-jrjst\" (UID: \"5d0c79f4-962d-4736-9953-2596e22ada2f\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-jrjst" Sep 09 11:07:17.575043 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:17.574892 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d0c79f4-962d-4736-9953-2596e22ada2f-lib-modules\") pod \"perf-node-gather-daemonset-jrjst\" (UID: \"5d0c79f4-962d-4736-9953-2596e22ada2f\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-jrjst" Sep 09 11:07:17.575043 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:17.574940 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8w46\" (UniqueName: \"kubernetes.io/projected/5d0c79f4-962d-4736-9953-2596e22ada2f-kube-api-access-x8w46\") pod \"perf-node-gather-daemonset-jrjst\" (UID: \"5d0c79f4-962d-4736-9953-2596e22ada2f\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-jrjst" Sep 09 11:07:17.676001 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:17.675957 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d0c79f4-962d-4736-9953-2596e22ada2f-sys\") pod \"perf-node-gather-daemonset-jrjst\" (UID: \"5d0c79f4-962d-4736-9953-2596e22ada2f\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-jrjst" Sep 09 11:07:17.676001 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:17.675998 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d0c79f4-962d-4736-9953-2596e22ada2f-lib-modules\") pod \"perf-node-gather-daemonset-jrjst\" (UID: \"5d0c79f4-962d-4736-9953-2596e22ada2f\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-jrjst" Sep 09 11:07:17.676251 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:17.676046 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8w46\" (UniqueName: \"kubernetes.io/projected/5d0c79f4-962d-4736-9953-2596e22ada2f-kube-api-access-x8w46\") pod \"perf-node-gather-daemonset-jrjst\" (UID: \"5d0c79f4-962d-4736-9953-2596e22ada2f\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-jrjst" Sep 09 11:07:17.676251 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:17.676082 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5d0c79f4-962d-4736-9953-2596e22ada2f-proc\") pod \"perf-node-gather-daemonset-jrjst\" (UID: \"5d0c79f4-962d-4736-9953-2596e22ada2f\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-jrjst" Sep 09 11:07:17.676251 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:17.676103 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5d0c79f4-962d-4736-9953-2596e22ada2f-podres\") pod \"perf-node-gather-daemonset-jrjst\" (UID: \"5d0c79f4-962d-4736-9953-2596e22ada2f\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-jrjst" Sep 09 11:07:17.676251 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:17.676112 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d0c79f4-962d-4736-9953-2596e22ada2f-sys\") pod \"perf-node-gather-daemonset-jrjst\" (UID: \"5d0c79f4-962d-4736-9953-2596e22ada2f\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-jrjst" Sep 09 11:07:17.676251 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:17.676204 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5d0c79f4-962d-4736-9953-2596e22ada2f-proc\") pod \"perf-node-gather-daemonset-jrjst\" (UID: \"5d0c79f4-962d-4736-9953-2596e22ada2f\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-jrjst" Sep 09 11:07:17.676251 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:17.676222 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d0c79f4-962d-4736-9953-2596e22ada2f-lib-modules\") pod \"perf-node-gather-daemonset-jrjst\" (UID: \"5d0c79f4-962d-4736-9953-2596e22ada2f\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-jrjst" Sep 09 11:07:17.676251 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:17.676222 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5d0c79f4-962d-4736-9953-2596e22ada2f-podres\") pod \"perf-node-gather-daemonset-jrjst\" (UID: \"5d0c79f4-962d-4736-9953-2596e22ada2f\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-jrjst" Sep 09 11:07:17.685676 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:17.685645 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8w46\" (UniqueName: \"kubernetes.io/projected/5d0c79f4-962d-4736-9953-2596e22ada2f-kube-api-access-x8w46\") pod \"perf-node-gather-daemonset-jrjst\" (UID: \"5d0c79f4-962d-4736-9953-2596e22ada2f\") " pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-jrjst" Sep 09 11:07:17.763669 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:17.763573 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-jrjst" Sep 09 11:07:18.045920 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:18.045892 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-86rrh/perf-node-gather-daemonset-jrjst"] Sep 09 11:07:18.050081 ip-10-0-139-133 kubenswrapper[2559]: W0909 11:07:18.050053 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5d0c79f4_962d_4736_9953_2596e22ada2f.slice/crio-8242ed9601c3b734baebf73fc33a6fcb7b061006f2a8a5f9d20b89487cfb32e7 WatchSource:0}: Error finding container 8242ed9601c3b734baebf73fc33a6fcb7b061006f2a8a5f9d20b89487cfb32e7: Status 404 returned error can't find the container with id 8242ed9601c3b734baebf73fc33a6fcb7b061006f2a8a5f9d20b89487cfb32e7 Sep 09 11:07:18.170063 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:18.170027 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-jrjst" event={"ID":"5d0c79f4-962d-4736-9953-2596e22ada2f","Type":"ContainerStarted","Data":"4eeb4c55579afa0530aa2c377e76580251139caedceb737936ad7d2e30e4a584"} Sep 09 11:07:18.170234 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:18.170082 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-jrjst" event={"ID":"5d0c79f4-962d-4736-9953-2596e22ada2f","Type":"ContainerStarted","Data":"8242ed9601c3b734baebf73fc33a6fcb7b061006f2a8a5f9d20b89487cfb32e7"} Sep 09 11:07:18.170234 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:18.170118 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-jrjst" Sep 09 11:07:18.190979 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:18.190924 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-jrjst" podStartSLOduration=1.190907411 podStartE2EDuration="1.190907411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 11:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 11:07:18.190318917 +0000 UTC m=+2478.364921242" watchObservedRunningTime="2026-09-09 11:07:18.190907411 +0000 UTC m=+2478.365509735" Sep 09 11:07:19.798843 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:19.798804 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-dhvgm_b19da99e-eafb-470c-ab9b-fe1e6800a92d/download-server/0.log" Sep 09 11:07:21.802888 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:21.802839 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jjw92_88316610-e58d-4f66-ae6c-eb81674b4f43/dns/0.log" Sep 09 11:07:21.833946 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:21.833905 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jjw92_88316610-e58d-4f66-ae6c-eb81674b4f43/kube-rbac-proxy/0.log" Sep 09 11:07:21.880791 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:21.880761 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4lsnr_13c59e77-04fc-4549-8402-df18b180161f/dns-node-resolver/0.log" Sep 09 11:07:22.639919 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:22.639880 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bq9w9_784872fd-624e-4323-862a-f1a040222214/node-ca/0.log" Sep 09 11:07:24.050395 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:24.050354 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-v2t7d_af4f8698-6435-4c45-8a43-0d2eba16ce43/discovery/0.log" Sep 09 11:07:24.131598 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:24.131569 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75cd99c5fc-fqj8r_c719a5cf-09c3-4950-916a-33922d99dad4/kube-auth-proxy/0.log" Sep 09 11:07:24.185110 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:24.185080 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-86rrh/perf-node-gather-daemonset-jrjst" Sep 09 11:07:24.345462 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:24.345433 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-x4njj_1cdfe780-23e6-4fa2-bb10-dd57f4c5c25b/payload-processing/0.log" Sep 09 11:07:25.533056 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:25.533029 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-llvb2_53bd305a-c566-47e2-bbfe-cddf73635d77/serve-healthcheck-canary/0.log" Sep 09 11:07:26.454428 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:26.454405 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rgrgx_44bad09f-4826-4024-8221-7abbd2051cba/kube-rbac-proxy/0.log" Sep 09 11:07:26.493272 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:26.493244 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rgrgx_44bad09f-4826-4024-8221-7abbd2051cba/exporter/0.log" Sep 09 11:07:26.529062 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:26.529037 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rgrgx_44bad09f-4826-4024-8221-7abbd2051cba/extractor/0.log" Sep 09 11:07:28.492088 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:28.492056 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-v2t7d_af4f8698-6435-4c45-8a43-0d2eba16ce43/discovery/0.log" Sep 09 11:07:28.539309 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:28.539282 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75cd99c5fc-fqj8r_c719a5cf-09c3-4950-916a-33922d99dad4/kube-auth-proxy/0.log" Sep 09 11:07:28.630929 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:28.630894 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-x4njj_1cdfe780-23e6-4fa2-bb10-dd57f4c5c25b/payload-processing/0.log" Sep 09 11:07:29.159179 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:29.159153 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-k45xt_2aa16cb0-c44e-48cd-986e-cc46caef5437/manager/0.log" Sep 09 11:07:29.359242 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:29.359214 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-xd7gg_3d352779-62ce-4e83-ac40-5de8a05026cc/manager/0.log" Sep 09 11:07:29.770676 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:29.770595 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-544cbf49fb-2nr65_a2658a6e-38ca-405b-bb5c-5767972f0fc9/manager/0.log" Sep 09 11:07:29.836543 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:29.836515 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-544cbf49fb-2nr65_a2658a6e-38ca-405b-bb5c-5767972f0fc9/copy-manifests/0.log" Sep 09 11:07:30.374948 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:30.374917 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-v2t7d_af4f8698-6435-4c45-8a43-0d2eba16ce43/discovery/0.log" Sep 09 11:07:30.442322 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:30.442283 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75cd99c5fc-fqj8r_c719a5cf-09c3-4950-916a-33922d99dad4/kube-auth-proxy/0.log" Sep 09 11:07:30.624434 ip-10-0-139-133 kubenswrapper[2559]: I0909 11:07:30.624401 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-x4njj_1cdfe780-23e6-4fa2-bb10-dd57f4c5c25b/payload-processing/0.log"